[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:56:05 dist-test-slave-j968 krb5kdc[31911](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:56:05 dist-test-slave-j968 krb5kdc[31911](info): set up 2 sockets
Sep 28 14:56:05 dist-test-slave-j968 krb5kdc[31911](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 14:56:06.501564 31904 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Sep 28 14:56:06 dist-test-slave-j968 krb5kdc[31911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:56:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:56:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab.
Entry for principal kudu/127.31.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab.
I20260928 14:56:06.688279 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:45431
--webserver_interface=127.31.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:44607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.40.62:45431
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:56:07.017951 31924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:56:07.018330 31924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:56:07.018510 31924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:56:07.025869 31924 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:56:07.026042 31924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:56:07.026129 31924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:56:07.026213 31924 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:56:07.026278 31924 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:56:07.036954 31924 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:44607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.40.62:45431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:45431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:56:07.038403 31924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:56:07.039772 31924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:56:07.052680 31932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:07.052672 31929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:07.052672 31930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:07.053064 31924 server_base.cc:1034] running on GCE node
I20260928 14:56:07.054378 31924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:56:07.056202 31924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:56:07.057437 31924 hybrid_clock.cc:648] HybridClock initialized: now 1790607367057385 us; error 36 us; skew 500 ppm
I20260928 14:56:07.057870 31924 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:56:07 dist-test-slave-j968 krb5kdc[31911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607367, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:56:07.061316 31924 init.cc:377] Logged in from keytab as kudu/127.31.40.62@KRBTEST.COM (short username kudu)
I20260928 14:56:07.062932 31924 webserver.cc:460] Webserver started at http://127.31.40.62:42923/ using document root <none> and password file <none>
I20260928 14:56:07.063616 31924 fs_manager.cc:359] Metadata directory not provided
I20260928 14:56:07.063712 31924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:56:07.064056 31924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:56:07.067143 31924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/master-0/data/instance:
uuid: "ef5db6f24e6246ec9b459693d0f5d085"
format_stamp: "Formatted at 2026-09-28 14:56:07 on dist-test-slave-j968"
server_key: "aa354522488373c9007dcf6ab9fff70e"
server_key_iv: "fe83665d49d46599ce4c8393b4ba2fe4"
server_key_version: "encryptionkey@0"
I20260928 14:56:07.068005 31924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/master-0/wal/instance:
uuid: "ef5db6f24e6246ec9b459693d0f5d085"
format_stamp: "Formatted at 2026-09-28 14:56:07 on dist-test-slave-j968"
server_key: "aa354522488373c9007dcf6ab9fff70e"
server_key_iv: "fe83665d49d46599ce4c8393b4ba2fe4"
server_key_version: "encryptionkey@0"
I20260928 14:56:07.073385 31924 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260928 14:56:07.077032 31939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:07.078474 31924 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 14:56:07.078769 31924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/master-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/master-0/wal
uuid: "ef5db6f24e6246ec9b459693d0f5d085"
format_stamp: "Formatted at 2026-09-28 14:56:07 on dist-test-slave-j968"
server_key: "aa354522488373c9007dcf6ab9fff70e"
server_key_iv: "fe83665d49d46599ce4c8393b4ba2fe4"
server_key_version: "encryptionkey@0"
I20260928 14:56:07.078986 31924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:56:07.106249 31924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:56:07.110435 31924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:56:07.110785 31924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:56:07.130003 31924 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.62:45431
I20260928 14:56:07.130028 31990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.62:45431 every 8 connection(s)
I20260928 14:56:07.131474 31924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
I20260928 14:56:07.135648 31991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:56:07.136425 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 31924
I20260928 14:56:07.136746 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/master-0/wal/instance
I20260928 14:56:07.138638 31904 external_mini_cluster.cc:1428] Setting key 801f6f0862a959e32a57e54093d5dd24
I20260928 14:56:07.147457 31991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef5db6f24e6246ec9b459693d0f5d085: Bootstrap starting.
I20260928 14:56:07.150681 31991 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef5db6f24e6246ec9b459693d0f5d085: Neither blocks nor log segments found. Creating new log.
I20260928 14:56:07.151729 31991 log.cc:824] T 00000000000000000000000000000000 P ef5db6f24e6246ec9b459693d0f5d085: Log is configured to *not* fsync() on all Append() calls
I20260928 14:56:07.154986 31991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef5db6f24e6246ec9b459693d0f5d085: No bootstrap required, opened a new log
I20260928 14:56:07.162374 31991 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef5db6f24e6246ec9b459693d0f5d085 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef5db6f24e6246ec9b459693d0f5d085" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 45431 } }
I20260928 14:56:07.162802 31991 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef5db6f24e6246ec9b459693d0f5d085 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:56:07.162910 31991 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef5db6f24e6246ec9b459693d0f5d085 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5db6f24e6246ec9b459693d0f5d085, State: Initialized, Role: FOLLOWER
I20260928 14:56:07.163511 31991 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef5db6f24e6246ec9b459693d0f5d085 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef5db6f24e6246ec9b459693d0f5d085" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 45431 } }
I20260928 14:56:07.163766 31991 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef5db6f24e6246ec9b459693d0f5d085 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:56:07.163947 31991 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef5db6f24e6246ec9b459693d0f5d085 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:56:07.164141 31991 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef5db6f24e6246ec9b459693d0f5d085 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:56:07.166301 31991 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef5db6f24e6246ec9b459693d0f5d085 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef5db6f24e6246ec9b459693d0f5d085" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 45431 } }
Sep 28 14:56:07 dist-test-slave-j968 krb5kdc[31911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:56:07.166852 31991 leader_election.cc:302] T 00000000000000000000000000000000 P ef5db6f24e6246ec9b459693d0f5d085 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef5db6f24e6246ec9b459693d0f5d085; no voters:
I20260928 14:56:07.167519 31991 leader_election.cc:288] T 00000000000000000000000000000000 P ef5db6f24e6246ec9b459693d0f5d085 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:56:07.167644 31996 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef5db6f24e6246ec9b459693d0f5d085 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:56:07.168913 31996 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef5db6f24e6246ec9b459693d0f5d085 [term 1 LEADER]: Becoming Leader. State: Replica: ef5db6f24e6246ec9b459693d0f5d085, State: Running, Role: LEADER
I20260928 14:56:07.169535 31996 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef5db6f24e6246ec9b459693d0f5d085 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef5db6f24e6246ec9b459693d0f5d085" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 45431 } }
I20260928 14:56:07.170228 31991 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef5db6f24e6246ec9b459693d0f5d085 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:56:07.173029 31997 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef5db6f24e6246ec9b459693d0f5d085 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef5db6f24e6246ec9b459693d0f5d085" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef5db6f24e6246ec9b459693d0f5d085" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 45431 } } }
I20260928 14:56:07.173527 31997 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef5db6f24e6246ec9b459693d0f5d085 [sys.catalog]: This master's current role is: LEADER
I20260928 14:56:07.173017 31998 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef5db6f24e6246ec9b459693d0f5d085 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef5db6f24e6246ec9b459693d0f5d085. Latest consensus state: current_term: 1 leader_uuid: "ef5db6f24e6246ec9b459693d0f5d085" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef5db6f24e6246ec9b459693d0f5d085" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 45431 } } }
I20260928 14:56:07.173871 31998 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef5db6f24e6246ec9b459693d0f5d085 [sys.catalog]: This master's current role is: LEADER
I20260928 14:56:07.178906 32002 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:56:07.181066 31993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:07.142637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56404 (local address 127.31.40.62:45431)
0928 14:56:07.143464 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:07.143475 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:07.144831 (+ 1356us) server_negotiation.cc:407] Connection header received
0928 14:56:07.154116 (+ 9285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:07.154137 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:07.154504 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:07.154898 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:07.157897 (+ 2999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:07.159262 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:07.161060 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:07.161742 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:07.161960 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:07.168144 (+ 6184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:07.168159 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:07.168186 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:07.168253 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:07.172332 (+ 4079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:07.173740 (+ 1408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:07.173752 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:07.173760 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:07.174175 (+ 415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:07.174980 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:07.174988 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:07.174998 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:07.175958 (+ 960us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:07.179213 (+ 3255us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:07.179427 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 14:56:07.179756 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":390,"threads_started":1}
I20260928 14:56:07.187408 32002 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:56:07.197134 32002 catalog_manager.cc:1347] Generated new cluster ID: 9a7f75255fae4912b8c839b654c12a65
I20260928 14:56:07.197273 32002 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:56:07.244647 32002 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:56:07.247076 32002 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:56:07.258016 32002 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef5db6f24e6246ec9b459693d0f5d085: Generated new TSK 0
I20260928 14:56:07.259279 32002 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.40.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab.
Entry for principal kudu/127.31.40.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab.
I20260928 14:56:07.301949 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.1:0
--local_ip_for_outbound_sockets=127.31.40.1
--webserver_interface=127.31.40.1
--webserver_port=0
--tserver_master_addrs=127.31.40.62:45431
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab
--principal=kudu/127.31.40.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:44607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:56:07.638450 32018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:56:07.638736 32018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:56:07.638830 32018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:56:07.646195 32018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:56:07.646323 32018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:56:07.646440 32018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.1
I20260928 14:56:07.656792 32018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:44607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab
--principal=kudu/127.31.40.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.40.1
--webserver_port=0
--tserver_master_addrs=127.31.40.62:45431
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.1
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:56:07.658154 32018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:56:07.659447 32018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:56:07.672899 32024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:07.673067 32023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:07.673357 32026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:07.674086 32018 server_base.cc:1034] running on GCE node
I20260928 14:56:07.674667 32018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:56:07.675585 32018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:56:07.676805 32018 hybrid_clock.cc:648] HybridClock initialized: now 1790607367676743 us; error 52 us; skew 500 ppm
I20260928 14:56:07.677162 32018 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:56:07 dist-test-slave-j968 krb5kdc[31911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607367, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:56:07.680565 32018 init.cc:377] Logged in from keytab as kudu/127.31.40.1@KRBTEST.COM (short username kudu)
I20260928 14:56:07.682238 32018 webserver.cc:460] Webserver started at http://127.31.40.1:38515/ using document root <none> and password file <none>
I20260928 14:56:07.682988 32018 fs_manager.cc:359] Metadata directory not provided
I20260928 14:56:07.683138 32018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:56:07.683454 32018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:56:07.686434 32018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/ts-0/data/instance:
uuid: "7a6708c940c0424689689355347e06dd"
format_stamp: "Formatted at 2026-09-28 14:56:07 on dist-test-slave-j968"
server_key: "ada5f8d0119f0ff6dbaf79fa54ca5afa"
server_key_iv: "c5015669af7d5ac09381adcf77985988"
server_key_version: "encryptionkey@0"
I20260928 14:56:07.687206 32018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/ts-0/wal/instance:
uuid: "7a6708c940c0424689689355347e06dd"
format_stamp: "Formatted at 2026-09-28 14:56:07 on dist-test-slave-j968"
server_key: "ada5f8d0119f0ff6dbaf79fa54ca5afa"
server_key_iv: "c5015669af7d5ac09381adcf77985988"
server_key_version: "encryptionkey@0"
I20260928 14:56:07.692359 32018 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 14:56:07.695744 32033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:07.697397 32018 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:56:07.697600 32018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/ts-0/wal
uuid: "7a6708c940c0424689689355347e06dd"
format_stamp: "Formatted at 2026-09-28 14:56:07 on dist-test-slave-j968"
server_key: "ada5f8d0119f0ff6dbaf79fa54ca5afa"
server_key_iv: "c5015669af7d5ac09381adcf77985988"
server_key_version: "encryptionkey@0"
I20260928 14:56:07.697839 32018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:56:07.719851 32018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:56:07.724437 32018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:56:07.724773 32018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:56:07.725838 32018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:56:07.727317 32018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:56:07.727435 32018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:07.727607 32018 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:56:07.727689 32018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:07.765172 32018 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.1:37265
I20260928 14:56:07.765216 32145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.1:37265 every 8 connection(s)
I20260928 14:56:07.766530 32018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
I20260928 14:56:07.770440 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 32018
I20260928 14:56:07.770711 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/ts-0/wal/instance
I20260928 14:56:07.771420 31904 external_mini_cluster.cc:1428] Setting key 878fd2fa3bb525dcf18553d07ee070d0
Sep 28 14:56:07 dist-test-slave-j968 krb5kdc[31911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607367, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.1@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.40.2@KRBTEST.COM; defaulting to no policy
I20260928 14:56:07.789685 32148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:07.769249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:46819 (local address 127.31.40.62:45431)
0928 14:56:07.769735 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:07.769740 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:07.771939 (+ 2199us) server_negotiation.cc:407] Connection header received
0928 14:56:07.773202 (+ 1263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:07.773207 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:07.773393 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:07.773559 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:07.775249 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:07.776075 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:07.778443 (+ 2368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:07.778858 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:07.779132 (+ 274us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:07.783647 (+ 4515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:07.783652 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:07.783655 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:07.783690 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:07.786435 (+ 2745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:07.787216 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:07.787224 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:07.787232 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:07.787347 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:07.787771 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:07.787777 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:07.787780 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:07.788125 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:07.788280 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:07.789048 (+ 768us) server_negotiation.cc:299] Negotiation successful
0928 14:56:07.789279 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":262,"threads_started":1}
I20260928 14:56:07.790791 32149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:07.769743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:45431 (local address 127.31.40.1:46819)
0928 14:56:07.770658 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:07.770699 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:07.772873 (+ 2174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:07.773870 (+ 997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:07.773879 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:07.774434 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:56:07.775060 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:07.775078 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:07.776260 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:07.776265 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:07.778278 (+ 2013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:07.778294 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:07.779228 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:07.779237 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:07.779597 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:07.780346 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:56:07.780388 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:56:07.783360 (+ 2972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:56:07.786598 (+ 3238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:07.786609 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:07.786628 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:07.787079 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:07.787465 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:07.787470 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:07.787474 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:07.787658 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:07.788252 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:56:07.788260 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:56:07.788620 (+ 360us) client_negotiation.cc:769] Sending connection context
0928 14:56:07.788947 (+ 327us) client_negotiation.cc:240] Negotiation successful
0928 14:56:07.789524 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":362,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.2@KRBTEST.COM" created.
I20260928 14:56:07.792459 32146 heartbeater.cc:347] Connected to a master server at 127.31.40.62:45431
I20260928 14:56:07.792799 32146 heartbeater.cc:464] Registering TS with master...
I20260928 14:56:07.793561 32146 heartbeater.cc:511] Master 127.31.40.62:45431 requested a full tablet report, sending...
I20260928 14:56:07.796209 31956 ts_manager.cc:194] Registered new tserver with Master: 7a6708c940c0424689689355347e06dd (127.31.40.1:37265)
I20260928 14:56:07.798513 31956 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.1@KRBTEST.COM'} at 127.31.40.1:46819
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab.
Entry for principal kudu/127.31.40.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab.
I20260928 14:56:07.815819 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.2:0
--local_ip_for_outbound_sockets=127.31.40.2
--webserver_interface=127.31.40.2
--webserver_port=0
--tserver_master_addrs=127.31.40.62:45431
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab
--principal=kudu/127.31.40.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:44607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:56:08.185946 32152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:56:08.186236 32152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:56:08.186426 32152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:56:08.193710 32152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:56:08.193822 32152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:56:08.194262 32152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.2
I20260928 14:56:08.204679 32152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:44607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab
--principal=kudu/127.31.40.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.40.2
--webserver_port=0
--tserver_master_addrs=127.31.40.62:45431
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.2
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:56:08.205960 32152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:56:08.207445 32152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:56:08.222126 32160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:08.222102 32158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:08.222230 32157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:08.223743 32152 server_base.cc:1034] running on GCE node
I20260928 14:56:08.224454 32152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:56:08.225519 32152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:56:08.226754 32152 hybrid_clock.cc:648] HybridClock initialized: now 1790607368226700 us; error 47 us; skew 500 ppm
I20260928 14:56:08.227095 32152 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:56:08 dist-test-slave-j968 krb5kdc[31911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607368, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:56:08.230307 32152 init.cc:377] Logged in from keytab as kudu/127.31.40.2@KRBTEST.COM (short username kudu)
I20260928 14:56:08.231716 32152 webserver.cc:460] Webserver started at http://127.31.40.2:34349/ using document root <none> and password file <none>
I20260928 14:56:08.232451 32152 fs_manager.cc:359] Metadata directory not provided
I20260928 14:56:08.232569 32152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:56:08.232884 32152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:56:08.235775 32152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/ts-1/data/instance:
uuid: "7e0dc7e1b01b4f3bbbdffc5d6d3c204f"
format_stamp: "Formatted at 2026-09-28 14:56:08 on dist-test-slave-j968"
server_key: "80568084869f045c2a68f45665fe6d10"
server_key_iv: "d3976344857b0c0843e80e5e29e53e55"
server_key_version: "encryptionkey@0"
I20260928 14:56:08.236603 32152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/ts-1/wal/instance:
uuid: "7e0dc7e1b01b4f3bbbdffc5d6d3c204f"
format_stamp: "Formatted at 2026-09-28 14:56:08 on dist-test-slave-j968"
server_key: "80568084869f045c2a68f45665fe6d10"
server_key_iv: "d3976344857b0c0843e80e5e29e53e55"
server_key_version: "encryptionkey@0"
I20260928 14:56:08.242000 32152 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 14:56:08.245767 32167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:08.247278 32152 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 14:56:08.247485 32152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/ts-1/wal
uuid: "7e0dc7e1b01b4f3bbbdffc5d6d3c204f"
format_stamp: "Formatted at 2026-09-28 14:56:08 on dist-test-slave-j968"
server_key: "80568084869f045c2a68f45665fe6d10"
server_key_iv: "d3976344857b0c0843e80e5e29e53e55"
server_key_version: "encryptionkey@0"
I20260928 14:56:08.247723 32152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:56:08.278110 32152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:56:08.283027 32152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:56:08.283324 32152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:56:08.284510 32152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:56:08.285846 32152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:56:08.285944 32152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:08.286046 32152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:56:08.286091 32152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:08.325220 32152 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.2:40101
I20260928 14:56:08.325270 32281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.2:40101 every 8 connection(s)
I20260928 14:56:08.326658 32152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
I20260928 14:56:08.334842 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 32152
I20260928 14:56:08.335088 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/ts-1/wal/instance
I20260928 14:56:08.335685 31904 external_mini_cluster.cc:1428] Setting key aa7caaaeacb52e760042de7c4fd4473a
Sep 28 14:56:08 dist-test-slave-j968 krb5kdc[31911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607368, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.2@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:56:08.350533 32284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:08.329425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:38395 (local address 127.31.40.62:45431)
0928 14:56:08.329884 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:08.329888 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:08.331027 (+ 1139us) server_negotiation.cc:407] Connection header received
0928 14:56:08.333127 (+ 2100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:08.333131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:08.333202 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:08.333302 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:08.335123 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:08.336239 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:08.338158 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:08.338885 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:08.339029 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:08.342770 (+ 3741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:08.342779 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:08.342786 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:08.342836 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:08.346587 (+ 3751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:08.347646 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:08.347650 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:08.347652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:08.347942 (+ 290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:08.348382 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:08.348389 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:08.348392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:08.348742 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:08.348943 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:08.349675 (+ 732us) server_negotiation.cc:299] Negotiation successful
0928 14:56:08.349904 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":219,"threads_started":1}
I20260928 14:56:08.351557 32285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:08.330015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:45431 (local address 127.31.40.2:38395)
0928 14:56:08.330817 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:08.330870 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:08.332835 (+ 1965us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:08.333587 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:08.333596 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:08.334163 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:56:08.334766 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:08.334784 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:08.336447 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:08.336453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:08.337992 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:08.338009 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:08.339139 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:08.339144 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:08.339353 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:08.340097 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:56:08.340121 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:56:08.342510 (+ 2389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:56:08.346746 (+ 4236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:08.346759 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:08.346783 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:08.347525 (+ 742us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:08.348063 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:08.348069 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:08.348072 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:08.348267 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:08.348885 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:56:08.348909 (+ 24us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:56:08.349296 (+ 387us) client_negotiation.cc:769] Sending connection context
0928 14:56:08.349589 (+ 293us) client_negotiation.cc:240] Negotiation successful
0928 14:56:08.350194 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":336,"threads_started":1}
WARNING: no policy specified for kudu/127.31.40.3@KRBTEST.COM; defaulting to no policy
I20260928 14:56:08.353539 32282 heartbeater.cc:347] Connected to a master server at 127.31.40.62:45431
I20260928 14:56:08.353967 32282 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.3@KRBTEST.COM" created.
I20260928 14:56:08.355194 32282 heartbeater.cc:511] Master 127.31.40.62:45431 requested a full tablet report, sending...
I20260928 14:56:08.357084 31956 ts_manager.cc:194] Registered new tserver with Master: 7e0dc7e1b01b4f3bbbdffc5d6d3c204f (127.31.40.2:40101)
I20260928 14:56:08.358366 31956 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.2@KRBTEST.COM'} at 127.31.40.2:38395
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab.
Entry for principal kudu/127.31.40.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab.
I20260928 14:56:08.380089 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.3:0
--local_ip_for_outbound_sockets=127.31.40.3
--webserver_interface=127.31.40.3
--webserver_port=0
--tserver_master_addrs=127.31.40.62:45431
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab
--principal=kudu/127.31.40.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:44607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:56:08.780936 32288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:56:08.781355 32288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:56:08.781541 32288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:56:08.789983 32288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:56:08.790150 32288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:56:08.790324 32288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.3
I20260928 14:56:08.800359 32288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:44607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab
--principal=kudu/127.31.40.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.40.3
--webserver_port=0
--tserver_master_addrs=127.31.40.62:45431
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.3
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:56:08.801769 32288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:56:08.801819 32146 heartbeater.cc:503] Master 127.31.40.62:45431 was elected leader, sending a full tablet report...
I20260928 14:56:08.803655 32288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:56:08.817704 32296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:08.817755 32293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:08.817714 32294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:08.818491 32288 server_base.cc:1034] running on GCE node
I20260928 14:56:08.819121 32288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:56:08.820199 32288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:56:08.821476 32288 hybrid_clock.cc:648] HybridClock initialized: now 1790607368821402 us; error 51 us; skew 500 ppm
I20260928 14:56:08.821844 32288 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:56:08 dist-test-slave-j968 krb5kdc[31911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607368, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:56:08.824970 32288 init.cc:377] Logged in from keytab as kudu/127.31.40.3@KRBTEST.COM (short username kudu)
I20260928 14:56:08.826470 32288 webserver.cc:460] Webserver started at http://127.31.40.3:39181/ using document root <none> and password file <none>
I20260928 14:56:08.827178 32288 fs_manager.cc:359] Metadata directory not provided
I20260928 14:56:08.827327 32288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:56:08.827661 32288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:56:08.830770 32288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/ts-2/data/instance:
uuid: "516a3900b9f94e7cbe67ebbaefe1137d"
format_stamp: "Formatted at 2026-09-28 14:56:08 on dist-test-slave-j968"
server_key: "63062ff257269e8901c37d1b22055fd6"
server_key_iv: "1617f0ba2f982a7edd491e0aadf239a1"
server_key_version: "encryptionkey@0"
I20260928 14:56:08.831526 32288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/ts-2/wal/instance:
uuid: "516a3900b9f94e7cbe67ebbaefe1137d"
format_stamp: "Formatted at 2026-09-28 14:56:08 on dist-test-slave-j968"
server_key: "63062ff257269e8901c37d1b22055fd6"
server_key_iv: "1617f0ba2f982a7edd491e0aadf239a1"
server_key_version: "encryptionkey@0"
I20260928 14:56:08.837097 32288 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 14:56:08.840997 32303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:08.842489 32288 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 14:56:08.842697 32288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/ts-2/wal
uuid: "516a3900b9f94e7cbe67ebbaefe1137d"
format_stamp: "Formatted at 2026-09-28 14:56:08 on dist-test-slave-j968"
server_key: "63062ff257269e8901c37d1b22055fd6"
server_key_iv: "1617f0ba2f982a7edd491e0aadf239a1"
server_key_version: "encryptionkey@0"
I20260928 14:56:08.842921 32288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:56:08.865720 32288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:56:08.870482 32288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:56:08.870806 32288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:56:08.871978 32288 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:56:08.873400 32288 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:56:08.873521 32288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:08.873685 32288 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:56:08.873737 32288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:08.909682 32288 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.3:36363
I20260928 14:56:08.909816 32416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.3:36363 every 8 connection(s)
I20260928 14:56:08.911502 32288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
I20260928 14:56:08.920832 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 32288
I20260928 14:56:08.921200 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607365362981-31904-0/minicluster-data/ts-2/wal/instance
I20260928 14:56:08.921891 31904 external_mini_cluster.cc:1428] Setting key 492c05d87d0cb4a32be95731082f75fc
Sep 28 14:56:08 dist-test-slave-j968 krb5kdc[31911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607368, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.3@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:56:08.936614 32419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:08.915483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:46451 (local address 127.31.40.62:45431)
0928 14:56:08.916355 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:08.916359 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:08.917874 (+ 1515us) server_negotiation.cc:407] Connection header received
0928 14:56:08.921411 (+ 3537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:08.921416 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:08.921546 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:08.921665 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:08.923930 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:08.925036 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:08.926642 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:08.927042 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:08.927184 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:08.930761 (+ 3577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:08.930776 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:08.930779 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:08.930822 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:08.933448 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:08.934218 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:08.934223 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:08.934227 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:08.934355 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:08.934793 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:08.934798 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:08.934802 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:08.935084 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:08.935246 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:08.935982 (+ 736us) server_negotiation.cc:299] Negotiation successful
0928 14:56:08.936211 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":737,"thread_start_us":570,"threads_started":1}
I20260928 14:56:08.937844 32420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:08.916441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:45431 (local address 127.31.40.3:46451)
0928 14:56:08.917689 (+ 1248us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:08.917739 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:08.920954 (+ 3215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:08.922206 (+ 1252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:08.922216 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:08.922947 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:56:08.923719 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:08.923739 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:08.925209 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:08.925213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:08.926474 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:08.926487 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:08.927276 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:08.927283 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:08.927528 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:08.928317 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:56:08.928341 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:56:08.930522 (+ 2181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:56:08.933617 (+ 3095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:08.933626 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:08.933646 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:08.934090 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:08.934467 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:08.934472 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:08.934475 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:08.934675 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:08.935211 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:56:08.935219 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:56:08.935590 (+ 371us) client_negotiation.cc:769] Sending connection context
0928 14:56:08.935866 (+ 276us) client_negotiation.cc:240] Negotiation successful
0928 14:56:08.936456 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":940,"thread_start_us":670,"threads_started":1}
I20260928 14:56:08.939486 32417 heartbeater.cc:347] Connected to a master server at 127.31.40.62:45431
I20260928 14:56:08.939755 32417 heartbeater.cc:464] Registering TS with master...
I20260928 14:56:08.940541 32417 heartbeater.cc:511] Master 127.31.40.62:45431 requested a full tablet report, sending...
I20260928 14:56:08.942282 31955 ts_manager.cc:194] Registered new tserver with Master: 516a3900b9f94e7cbe67ebbaefe1137d (127.31.40.3:36363)
I20260928 14:56:08.943418 31955 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.3@KRBTEST.COM'} at 127.31.40.3:46451
I20260928 14:56:08.951694 31904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 14:56:08 dist-test-slave-j968 krb5kdc[31911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607368, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 14:56:08 dist-test-slave-j968 krb5kdc[31911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607368, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.40.1@KRBTEST.COM
I20260928 14:56:08.999500 32424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:08.984070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41684 (local address 127.31.40.1:37265)
0928 14:56:08.984695 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:08.984703 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:08.985196 (+ 493us) server_negotiation.cc:407] Connection header received
0928 14:56:08.985332 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:08.985340 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:08.985593 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:08.985729 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:08.987201 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:08.988290 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:08.989547 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:08.990108 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:08.990213 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:08.994202 (+ 3989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:08.994215 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:08.994231 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:08.994265 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:08.997020 (+ 2755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:08.997664 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:08.997671 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:08.997674 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:08.997811 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:08.998175 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:08.998179 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:08.998181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:08.998517 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:08.998757 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:08.998969 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 14:56:08.999172 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":329,"threads_started":1}
W20260928 14:56:09.000885 32061 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41684
Sep 28 14:56:09 dist-test-slave-j968 krb5kdc[31911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 14:56:09 dist-test-slave-j968 krb5kdc[31911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.1@KRBTEST.COM
I20260928 14:56:09.045647 32424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:09.031469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41694 (local address 127.31.40.1:37265)
0928 14:56:09.031709 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:09.031717 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:09.031925 (+ 208us) server_negotiation.cc:407] Connection header received
0928 14:56:09.032116 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:09.032121 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:09.032212 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:09.032299 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:09.033861 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:09.034562 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:09.035917 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:09.036349 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:09.036446 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:09.040418 (+ 3972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:09.040423 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:09.040427 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:09.040462 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:09.043097 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:09.043786 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:09.043790 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:09.043792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:09.043958 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:09.044451 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:09.044454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:09.044456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:09.044706 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:09.044887 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:09.045216 (+ 329us) server_negotiation.cc:299] Negotiation successful
0928 14:56:09.045376 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260928 14:56:09.047382 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 32018
I20260928 14:56:09.061905 32424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:09.048449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41706 (local address 127.31.40.1:37265)
0928 14:56:09.048686 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:09.048694 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:09.048981 (+ 287us) server_negotiation.cc:407] Connection header received
0928 14:56:09.049191 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:09.049197 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:09.049270 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:09.049370 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:09.050709 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:09.051543 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:09.053091 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:09.053887 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:09.054043 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:09.056492 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:09.056497 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:09.056500 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:09.056531 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:09.059048 (+ 2517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:09.059712 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:09.059716 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:09.059718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:09.059850 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:09.060247 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:09.060255 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:09.060261 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:09.060640 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:09.060900 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:09.061091 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 14:56:09.061373 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 14:56:09.062709 32141 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:56:09.147048 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 32018
I20260928 14:56:09.163982 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 32152
Sep 28 14:56:09 dist-test-slave-j968 krb5kdc[31911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.2@KRBTEST.COM
I20260928 14:56:09.185182 32432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:09.165025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40758 (local address 127.31.40.2:40101)
0928 14:56:09.165641 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:09.165649 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:09.165675 (+ 26us) server_negotiation.cc:407] Connection header received
0928 14:56:09.165736 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:09.165742 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:09.165967 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:09.166097 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:09.167256 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:09.168389 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:09.169573 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:09.170145 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:09.170248 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:09.178624 (+ 8376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:09.178640 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:09.178657 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:09.178701 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:09.182084 (+ 3383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:09.182765 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:09.182772 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:09.182774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:09.183012 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:09.183408 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:09.183411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:09.183413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:09.183775 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:09.184061 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:09.184445 (+ 384us) server_negotiation.cc:299] Negotiation successful
0928 14:56:09.184760 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":309,"threads_started":1}
I20260928 14:56:09.186497 32277 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:56:09.269876 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 32152
I20260928 14:56:09.286584 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 32288
Sep 28 14:56:09 dist-test-slave-j968 krb5kdc[31911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.3@KRBTEST.COM
I20260928 14:56:09.303225 32434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:09.287672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43848 (local address 127.31.40.3:36363)
0928 14:56:09.288507 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:09.288514 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:09.288544 (+ 30us) server_negotiation.cc:407] Connection header received
0928 14:56:09.288606 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:09.288612 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:09.288833 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:09.288974 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:09.290125 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:09.291276 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:09.292569 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:09.293234 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:09.293341 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:09.297526 (+ 4185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:09.297537 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:09.297553 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:09.297592 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:09.300705 (+ 3113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:09.301288 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:09.301295 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:09.301297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:09.301423 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:09.301780 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:09.301784 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:09.301787 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:09.302203 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:09.302434 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:09.302708 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 14:56:09.302908 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":495,"threads_started":1}
I20260928 14:56:09.304612 32412 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:56:09.392089 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 32288
I20260928 14:56:09.408723 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 31924
I20260928 14:56:09.409951 31986 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:56:09.515043 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 31924
2026-09-28T14:56:09Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4117 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:56:09 dist-test-slave-j968 krb5kdc[32439](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:56:09 dist-test-slave-j968 krb5kdc[32439](info): set up 2 sockets
Sep 28 14:56:09 dist-test-slave-j968 krb5kdc[32439](info): commencing operation
krb5kdc: starting...
W20260928 14:56:10.629750 31904 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.004s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Sep 28 14:56:10 dist-test-slave-j968 krb5kdc[32439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:56:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:56:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab.
Entry for principal kudu/127.31.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab.
I20260928 14:56:10.824064 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:43337
--webserver_interface=127.31.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:46481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.40.62:43337
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:56:11.158015 32452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:56:11.158399 32452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:56:11.158571 32452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:56:11.166023 32452 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:56:11.166215 32452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:56:11.166340 32452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:56:11.166441 32452 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:56:11.166529 32452 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:56:11.176827 32452 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:46481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.40.62:43337
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:43337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:56:11.178323 32452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:56:11.179719 32452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:56:11.193051 32457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:11.193105 32460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:11.193130 32458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:11.194916 32452 server_base.cc:1034] running on GCE node
I20260928 14:56:11.195827 32452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:56:11.197520 32452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:56:11.198783 32452 hybrid_clock.cc:648] HybridClock initialized: now 1790607371198737 us; error 32 us; skew 500 ppm
I20260928 14:56:11.199220 32452 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:56:11 dist-test-slave-j968 krb5kdc[32439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607371, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:56:11.202692 32452 init.cc:377] Logged in from keytab as kudu/127.31.40.62@KRBTEST.COM (short username kudu)
I20260928 14:56:11.204522 32452 webserver.cc:460] Webserver started at http://127.31.40.62:34327/ using document root <none> and password file <none>
I20260928 14:56:11.205312 32452 fs_manager.cc:359] Metadata directory not provided
I20260928 14:56:11.205471 32452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:56:11.205817 32452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:56:11.209179 32452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/master-0/data/instance:
uuid: "326e6663aa37468bb9dee8763da08eb6"
format_stamp: "Formatted at 2026-09-28 14:56:11 on dist-test-slave-j968"
server_key: "e5d11c4d7084d61079aeb89e8ecc5959"
server_key_iv: "63d773a1a0d810d0b6c0c172dc027a9f"
server_key_version: "encryptionkey@0"
I20260928 14:56:11.210036 32452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/master-0/wal/instance:
uuid: "326e6663aa37468bb9dee8763da08eb6"
format_stamp: "Formatted at 2026-09-28 14:56:11 on dist-test-slave-j968"
server_key: "e5d11c4d7084d61079aeb89e8ecc5959"
server_key_iv: "63d773a1a0d810d0b6c0c172dc027a9f"
server_key_version: "encryptionkey@0"
I20260928 14:56:11.216212 32452 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260928 14:56:11.220572 32467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:11.222349 32452 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 14:56:11.222579 32452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/master-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/master-0/wal
uuid: "326e6663aa37468bb9dee8763da08eb6"
format_stamp: "Formatted at 2026-09-28 14:56:11 on dist-test-slave-j968"
server_key: "e5d11c4d7084d61079aeb89e8ecc5959"
server_key_iv: "63d773a1a0d810d0b6c0c172dc027a9f"
server_key_version: "encryptionkey@0"
I20260928 14:56:11.222836 32452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:56:11.243235 32452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:56:11.247941 32452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:56:11.248380 32452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:56:11.269446 32452 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.62:43337
I20260928 14:56:11.269510 32519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.62:43337 every 8 connection(s)
I20260928 14:56:11.271027 32452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
I20260928 14:56:11.272984 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 32452
I20260928 14:56:11.273245 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/master-0/wal/instance
I20260928 14:56:11.274116 31904 external_mini_cluster.cc:1428] Setting key cffb36675aaefc3a538492b4a4e67373
I20260928 14:56:11.276206 32520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 14:56:11 dist-test-slave-j968 krb5kdc[32439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:56:11.290486 32520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 326e6663aa37468bb9dee8763da08eb6: Bootstrap starting.
I20260928 14:56:11.293864 32520 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 326e6663aa37468bb9dee8763da08eb6: Neither blocks nor log segments found. Creating new log.
I20260928 14:56:11.294974 32520 log.cc:824] T 00000000000000000000000000000000 P 326e6663aa37468bb9dee8763da08eb6: Log is configured to *not* fsync() on all Append() calls
I20260928 14:56:11.298055 32520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 326e6663aa37468bb9dee8763da08eb6: No bootstrap required, opened a new log
I20260928 14:56:11.299247 32523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:11.276418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41486 (local address 127.31.40.62:43337)
0928 14:56:11.277288 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:11.277301 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:11.277343 (+ 42us) server_negotiation.cc:407] Connection header received
0928 14:56:11.279065 (+ 1722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:11.279088 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:11.279522 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:11.279962 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:11.281771 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:11.283336 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:11.285071 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:11.285854 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:11.286006 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:11.290934 (+ 4928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:11.290942 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:11.290960 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:11.291006 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:11.294589 (+ 3583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:11.295435 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:11.295443 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:11.295450 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:11.295642 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:11.296174 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:11.296178 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:11.296181 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:11.296760 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:11.297102 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:11.297632 (+ 530us) server_negotiation.cc:299] Negotiation successful
0928 14:56:11.298096 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":399,"threads_started":1}
I20260928 14:56:11.307044 32520 raft_consensus.cc:348] T 00000000000000000000000000000000 P 326e6663aa37468bb9dee8763da08eb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "326e6663aa37468bb9dee8763da08eb6" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 43337 } }
I20260928 14:56:11.307482 32520 raft_consensus.cc:374] T 00000000000000000000000000000000 P 326e6663aa37468bb9dee8763da08eb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:56:11.307602 32520 raft_consensus.cc:729] T 00000000000000000000000000000000 P 326e6663aa37468bb9dee8763da08eb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 326e6663aa37468bb9dee8763da08eb6, State: Initialized, Role: FOLLOWER
I20260928 14:56:11.308427 32520 consensus_queue.cc:260] T 00000000000000000000000000000000 P 326e6663aa37468bb9dee8763da08eb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "326e6663aa37468bb9dee8763da08eb6" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 43337 } }
I20260928 14:56:11.308696 32520 raft_consensus.cc:388] T 00000000000000000000000000000000 P 326e6663aa37468bb9dee8763da08eb6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:56:11.308844 32520 raft_consensus.cc:482] T 00000000000000000000000000000000 P 326e6663aa37468bb9dee8763da08eb6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:56:11.309038 32520 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 326e6663aa37468bb9dee8763da08eb6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:56:11.311973 32520 raft_consensus.cc:504] T 00000000000000000000000000000000 P 326e6663aa37468bb9dee8763da08eb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "326e6663aa37468bb9dee8763da08eb6" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 43337 } }
I20260928 14:56:11.312572 32520 leader_election.cc:302] T 00000000000000000000000000000000 P 326e6663aa37468bb9dee8763da08eb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 326e6663aa37468bb9dee8763da08eb6; no voters:
I20260928 14:56:11.313235 32520 leader_election.cc:288] T 00000000000000000000000000000000 P 326e6663aa37468bb9dee8763da08eb6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:56:11.313409 32525 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 326e6663aa37468bb9dee8763da08eb6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:56:11.314127 32525 raft_consensus.cc:686] T 00000000000000000000000000000000 P 326e6663aa37468bb9dee8763da08eb6 [term 1 LEADER]: Becoming Leader. State: Replica: 326e6663aa37468bb9dee8763da08eb6, State: Running, Role: LEADER
I20260928 14:56:11.314764 32525 consensus_queue.cc:237] T 00000000000000000000000000000000 P 326e6663aa37468bb9dee8763da08eb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "326e6663aa37468bb9dee8763da08eb6" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 43337 } }
I20260928 14:56:11.315732 32520 sys_catalog.cc:564] T 00000000000000000000000000000000 P 326e6663aa37468bb9dee8763da08eb6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:56:11.317670 32526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 326e6663aa37468bb9dee8763da08eb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "326e6663aa37468bb9dee8763da08eb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "326e6663aa37468bb9dee8763da08eb6" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 43337 } } }
I20260928 14:56:11.318187 32526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 326e6663aa37468bb9dee8763da08eb6 [sys.catalog]: This master's current role is: LEADER
I20260928 14:56:11.318523 32527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 326e6663aa37468bb9dee8763da08eb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 326e6663aa37468bb9dee8763da08eb6. Latest consensus state: current_term: 1 leader_uuid: "326e6663aa37468bb9dee8763da08eb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "326e6663aa37468bb9dee8763da08eb6" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 43337 } } }
I20260928 14:56:11.318894 32527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 326e6663aa37468bb9dee8763da08eb6 [sys.catalog]: This master's current role is: LEADER
I20260928 14:56:11.320724 32531 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:56:11.326294 32531 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:56:11.337920 32531 catalog_manager.cc:1347] Generated new cluster ID: c56c7a101b234a198a210e9d147aa862
I20260928 14:56:11.338064 32531 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:56:11.362468 32531 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:56:11.364781 32531 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:56:11.376266 32531 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 326e6663aa37468bb9dee8763da08eb6: Generated new TSK 0
I20260928 14:56:11.377553 32531 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.40.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab.
Entry for principal kudu/127.31.40.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab.
I20260928 14:56:11.431645 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.1:0
--local_ip_for_outbound_sockets=127.31.40.1
--webserver_interface=127.31.40.1
--webserver_port=0
--tserver_master_addrs=127.31.40.62:43337
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab
--principal=kudu/127.31.40.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:46481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:56:11.778680 32546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:56:11.779001 32546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:56:11.779100 32546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:56:11.786576 32546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:56:11.786701 32546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:56:11.786818 32546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.1
I20260928 14:56:11.797108 32546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:46481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab
--principal=kudu/127.31.40.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.40.1
--webserver_port=0
--tserver_master_addrs=127.31.40.62:43337
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.1
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:56:11.798431 32546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:56:11.799803 32546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:56:11.813596 32551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:11.813517 32552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:11.813573 32554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:11.814330 32546 server_base.cc:1034] running on GCE node
I20260928 14:56:11.814975 32546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:56:11.816098 32546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:56:11.817328 32546 hybrid_clock.cc:648] HybridClock initialized: now 1790607371817274 us; error 53 us; skew 500 ppm
I20260928 14:56:11.817703 32546 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:56:11 dist-test-slave-j968 krb5kdc[32439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607371, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:56:11.821221 32546 init.cc:377] Logged in from keytab as kudu/127.31.40.1@KRBTEST.COM (short username kudu)
I20260928 14:56:11.822734 32546 webserver.cc:460] Webserver started at http://127.31.40.1:42845/ using document root <none> and password file <none>
I20260928 14:56:11.823470 32546 fs_manager.cc:359] Metadata directory not provided
I20260928 14:56:11.823640 32546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:56:11.824048 32546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:56:11.827390 32546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/ts-0/data/instance:
uuid: "2c6a384011f1414f9fc0933fe3fc66c6"
format_stamp: "Formatted at 2026-09-28 14:56:11 on dist-test-slave-j968"
server_key: "51ff2a5f34fe5a6090cca7f8371ca41a"
server_key_iv: "d412e53147e090946987abf5edbd85f6"
server_key_version: "encryptionkey@0"
I20260928 14:56:11.828346 32546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/ts-0/wal/instance:
uuid: "2c6a384011f1414f9fc0933fe3fc66c6"
format_stamp: "Formatted at 2026-09-28 14:56:11 on dist-test-slave-j968"
server_key: "51ff2a5f34fe5a6090cca7f8371ca41a"
server_key_iv: "d412e53147e090946987abf5edbd85f6"
server_key_version: "encryptionkey@0"
I20260928 14:56:11.833981 32546 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 14:56:11.837615 32561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:11.839010 32546 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 14:56:11.839203 32546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/ts-0/wal
uuid: "2c6a384011f1414f9fc0933fe3fc66c6"
format_stamp: "Formatted at 2026-09-28 14:56:11 on dist-test-slave-j968"
server_key: "51ff2a5f34fe5a6090cca7f8371ca41a"
server_key_iv: "d412e53147e090946987abf5edbd85f6"
server_key_version: "encryptionkey@0"
I20260928 14:56:11.839416 32546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:56:11.851068 32546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:56:11.856122 32546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:56:11.856454 32546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:56:11.857604 32546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:56:11.859081 32546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:56:11.859195 32546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:11.859352 32546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:56:11.859428 32546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:11.893695 32546 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.1:41391
I20260928 14:56:11.893743 32674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.1:41391 every 8 connection(s)
I20260928 14:56:11.895028 32546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
I20260928 14:56:11.900193 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 32546
I20260928 14:56:11.900429 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/ts-0/wal/instance
I20260928 14:56:11.901039 31904 external_mini_cluster.cc:1428] Setting key 7bd500751ed4704abae68dd21d368e30
Sep 28 14:56:11 dist-test-slave-j968 krb5kdc[32439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607371, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.1@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.40.2@KRBTEST.COM; defaulting to no policy
I20260928 14:56:11.918309 32677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:11.897680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:51193 (local address 127.31.40.62:43337)
0928 14:56:11.898109 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:11.898116 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:11.899194 (+ 1078us) server_negotiation.cc:407] Connection header received
0928 14:56:11.901459 (+ 2265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:11.901465 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:11.901590 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:11.901713 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:11.903496 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:11.904748 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:11.906478 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:11.907219 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:11.907365 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:11.911821 (+ 4456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:11.911826 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:11.911829 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:11.911903 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:11.914614 (+ 2711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:11.915392 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:11.915398 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:11.915400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:11.915519 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:11.916052 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:11.916058 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:11.916061 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:11.916462 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:11.916624 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:11.917745 (+ 1121us) server_negotiation.cc:299] Negotiation successful
0928 14:56:11.917949 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":195,"threads_started":1}
I20260928 14:56:11.919486 32678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:11.898323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43337 (local address 127.31.40.1:51193)
0928 14:56:11.899025 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:11.899063 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:11.901126 (+ 2063us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:11.902033 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:11.902042 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:11.902560 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:56:11.903291 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:11.903310 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:11.904917 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:11.904921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:11.906313 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:11.906331 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:11.907465 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:11.907474 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:11.907844 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:11.908704 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:56:11.908742 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:56:11.911575 (+ 2833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:56:11.914762 (+ 3187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:11.914772 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:11.914792 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:11.915249 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:11.915667 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:11.915672 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:11.915676 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:11.915934 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:11.916662 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:56:11.916675 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:56:11.917208 (+ 533us) client_negotiation.cc:769] Sending connection context
0928 14:56:11.917581 (+ 373us) client_negotiation.cc:240] Negotiation successful
0928 14:56:11.918215 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":282,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.2@KRBTEST.COM" created.
I20260928 14:56:11.921381 32675 heartbeater.cc:347] Connected to a master server at 127.31.40.62:43337
I20260928 14:56:11.921797 32675 heartbeater.cc:464] Registering TS with master...
I20260928 14:56:11.922828 32675 heartbeater.cc:511] Master 127.31.40.62:43337 requested a full tablet report, sending...
I20260928 14:56:11.925324 32484 ts_manager.cc:194] Registered new tserver with Master: 2c6a384011f1414f9fc0933fe3fc66c6 (127.31.40.1:41391)
I20260928 14:56:11.927755 32484 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.1@KRBTEST.COM'} at 127.31.40.1:51193
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab.
Entry for principal kudu/127.31.40.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab.
I20260928 14:56:11.947705 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.2:0
--local_ip_for_outbound_sockets=127.31.40.2
--webserver_interface=127.31.40.2
--webserver_port=0
--tserver_master_addrs=127.31.40.62:43337
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab
--principal=kudu/127.31.40.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:46481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:56:12.284206 32681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:56:12.284490 32681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:56:12.284669 32681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:56:12.292201 32681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:56:12.292330 32681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:56:12.292565 32681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.2
I20260928 14:56:12.302678 32681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:46481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab
--principal=kudu/127.31.40.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.40.2
--webserver_port=0
--tserver_master_addrs=127.31.40.62:43337
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.2
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:56:12.304170 32681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:56:12.305572 32681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:56:12.319741 32686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:12.319700 32687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:12.319761 32689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:12.325707 32681 server_base.cc:1034] running on GCE node
I20260928 14:56:12.326650 32681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:56:12.328202 32681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:56:12.329504 32681 hybrid_clock.cc:648] HybridClock initialized: now 1790607372329420 us; error 66 us; skew 500 ppm
I20260928 14:56:12.329871 32681 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:56:12 dist-test-slave-j968 krb5kdc[32439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607372, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:56:12.333503 32681 init.cc:377] Logged in from keytab as kudu/127.31.40.2@KRBTEST.COM (short username kudu)
I20260928 14:56:12.335223 32681 webserver.cc:460] Webserver started at http://127.31.40.2:37359/ using document root <none> and password file <none>
I20260928 14:56:12.336091 32681 fs_manager.cc:359] Metadata directory not provided
I20260928 14:56:12.336242 32681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:56:12.336563 32681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:56:12.339537 32681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/ts-1/data/instance:
uuid: "8f56ad9a62474242bb63993a297328d6"
format_stamp: "Formatted at 2026-09-28 14:56:12 on dist-test-slave-j968"
server_key: "66e4e5b4022337f4cb22cbeb0afc5b3e"
server_key_iv: "fd00ffe1fefdc0f880b34d9289646949"
server_key_version: "encryptionkey@0"
I20260928 14:56:12.340458 32681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/ts-1/wal/instance:
uuid: "8f56ad9a62474242bb63993a297328d6"
format_stamp: "Formatted at 2026-09-28 14:56:12 on dist-test-slave-j968"
server_key: "66e4e5b4022337f4cb22cbeb0afc5b3e"
server_key_iv: "fd00ffe1fefdc0f880b34d9289646949"
server_key_version: "encryptionkey@0"
I20260928 14:56:12.346058 32681 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 14:56:12.349879 32696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:12.351387 32681 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:56:12.351580 32681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/ts-1/wal
uuid: "8f56ad9a62474242bb63993a297328d6"
format_stamp: "Formatted at 2026-09-28 14:56:12 on dist-test-slave-j968"
server_key: "66e4e5b4022337f4cb22cbeb0afc5b3e"
server_key_iv: "fd00ffe1fefdc0f880b34d9289646949"
server_key_version: "encryptionkey@0"
I20260928 14:56:12.351806 32681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:56:12.384441 32681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:56:12.388938 32681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:56:12.389252 32681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:56:12.390301 32681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:56:12.391992 32681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:56:12.392098 32681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:12.392234 32681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:56:12.392298 32681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:12.434386 32681 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.2:38065
I20260928 14:56:12.434427 341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.2:38065 every 8 connection(s)
I20260928 14:56:12.435745 32681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
I20260928 14:56:12.436872 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 32681
I20260928 14:56:12.437077 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/ts-1/wal/instance
I20260928 14:56:12.437498 31904 external_mini_cluster.cc:1428] Setting key 4ccecf9e28091ddee108e1c120d67114
Sep 28 14:56:12 dist-test-slave-j968 krb5kdc[32439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607372, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.2@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.40.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.3@KRBTEST.COM" created.
I20260928 14:56:12.458763 344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:12.438655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:50703 (local address 127.31.40.62:43337)
0928 14:56:12.439119 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:12.439126 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:12.440218 (+ 1092us) server_negotiation.cc:407] Connection header received
0928 14:56:12.442070 (+ 1852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:12.442075 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:12.442169 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:12.442286 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:12.443995 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:12.444720 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:12.447359 (+ 2639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:12.447799 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:12.447999 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:12.452470 (+ 4471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:12.452475 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:12.452478 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:12.452512 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:12.455385 (+ 2873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:12.456294 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:12.456299 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:12.456301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:12.456435 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:12.456896 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:12.456899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:12.456901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:12.457200 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:12.457338 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:12.458128 (+ 790us) server_negotiation.cc:299] Negotiation successful
0928 14:56:12.458359 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":226,"threads_started":1}
I20260928 14:56:12.459739 345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:12.439235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43337 (local address 127.31.40.2:50703)
0928 14:56:12.440065 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:12.440108 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:12.441784 (+ 1676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:12.442560 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:12.442569 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:12.443112 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:56:12.443780 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:12.443810 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:12.444878 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:12.444886 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:12.447192 (+ 2306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:12.447211 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:12.448127 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:12.448137 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:12.448468 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:12.449278 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:56:12.449314 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:56:12.452208 (+ 2894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:56:12.455544 (+ 3336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:12.455562 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:12.455583 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:12.456132 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:12.456522 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:12.456525 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:12.456528 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:12.456779 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:12.457323 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:56:12.457330 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:56:12.457708 (+ 378us) client_negotiation.cc:769] Sending connection context
0928 14:56:12.458022 (+ 314us) client_negotiation.cc:240] Negotiation successful
0928 14:56:12.458583 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":352,"threads_started":1}
I20260928 14:56:12.461202 342 heartbeater.cc:347] Connected to a master server at 127.31.40.62:43337
I20260928 14:56:12.461544 342 heartbeater.cc:464] Registering TS with master...
I20260928 14:56:12.462453 342 heartbeater.cc:511] Master 127.31.40.62:43337 requested a full tablet report, sending...
I20260928 14:56:12.464303 32484 ts_manager.cc:194] Registered new tserver with Master: 8f56ad9a62474242bb63993a297328d6 (127.31.40.2:38065)
I20260928 14:56:12.465170 32484 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.2@KRBTEST.COM'} at 127.31.40.2:50703
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab.
Entry for principal kudu/127.31.40.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab.
I20260928 14:56:12.482831 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.3:0
--local_ip_for_outbound_sockets=127.31.40.3
--webserver_interface=127.31.40.3
--webserver_port=0
--tserver_master_addrs=127.31.40.62:43337
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab
--principal=kudu/127.31.40.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:46481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:56:12.832420 348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:56:12.832741 348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:56:12.832849 348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:56:12.840878 348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:56:12.841004 348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:56:12.841126 348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.3
I20260928 14:56:12.851394 348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:46481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab
--principal=kudu/127.31.40.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.40.3
--webserver_port=0
--tserver_master_addrs=127.31.40.62:43337
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.3
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:56:12.852888 348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:56:12.854256 348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:56:12.869289 354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:12.869307 353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:12.869575 356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:12.870594 348 server_base.cc:1034] running on GCE node
I20260928 14:56:12.871294 348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:56:12.872416 348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:56:12.873660 348 hybrid_clock.cc:648] HybridClock initialized: now 1790607372873597 us; error 45 us; skew 500 ppm
I20260928 14:56:12.874166 348 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:56:12 dist-test-slave-j968 krb5kdc[32439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607372, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:56:12.878122 348 init.cc:377] Logged in from keytab as kudu/127.31.40.3@KRBTEST.COM (short username kudu)
I20260928 14:56:12.880095 348 webserver.cc:460] Webserver started at http://127.31.40.3:40155/ using document root <none> and password file <none>
I20260928 14:56:12.880905 348 fs_manager.cc:359] Metadata directory not provided
I20260928 14:56:12.881090 348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:56:12.881459 348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:56:12.884874 348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/ts-2/data/instance:
uuid: "e6f6dc3b2d2b46c4b068562b1568c2db"
format_stamp: "Formatted at 2026-09-28 14:56:12 on dist-test-slave-j968"
server_key: "8c2228d455a5132e14d07a33f2335710"
server_key_iv: "88092a1ad904256b04a0745d0c07c4da"
server_key_version: "encryptionkey@0"
I20260928 14:56:12.885785 348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/ts-2/wal/instance:
uuid: "e6f6dc3b2d2b46c4b068562b1568c2db"
format_stamp: "Formatted at 2026-09-28 14:56:12 on dist-test-slave-j968"
server_key: "8c2228d455a5132e14d07a33f2335710"
server_key_iv: "88092a1ad904256b04a0745d0c07c4da"
server_key_version: "encryptionkey@0"
I20260928 14:56:12.892052 348 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.002s
I20260928 14:56:12.895955 363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:12.897469 348 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:56:12.897634 348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/ts-2/wal
uuid: "e6f6dc3b2d2b46c4b068562b1568c2db"
format_stamp: "Formatted at 2026-09-28 14:56:12 on dist-test-slave-j968"
server_key: "8c2228d455a5132e14d07a33f2335710"
server_key_iv: "88092a1ad904256b04a0745d0c07c4da"
server_key_version: "encryptionkey@0"
I20260928 14:56:12.897827 348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:56:12.914741 348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:56:12.919515 348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:56:12.919806 348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:56:12.920925 348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:56:12.922358 348 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:56:12.922431 348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:12.922585 348 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:56:12.922636 348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:12.932024 32675 heartbeater.cc:503] Master 127.31.40.62:43337 was elected leader, sending a full tablet report...
I20260928 14:56:12.958381 348 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.3:33741
I20260928 14:56:12.958359 478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.3:33741 every 8 connection(s)
I20260928 14:56:12.959743 348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
I20260928 14:56:12.963229 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 348
I20260928 14:56:12.963460 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607365362981-31904-0/minicluster-data/ts-2/wal/instance
I20260928 14:56:12.964052 31904 external_mini_cluster.cc:1428] Setting key a60802fe7f8f39043efa5019d8197d3a
Sep 28 14:56:12 dist-test-slave-j968 krb5kdc[32439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607372, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.3@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:56:12.981742 481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:12.962502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:52427 (local address 127.31.40.62:43337)
0928 14:56:12.963000 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:12.963005 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:12.964188 (+ 1183us) server_negotiation.cc:407] Connection header received
0928 14:56:12.966287 (+ 2099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:12.966292 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:12.966399 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:12.966511 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:12.968275 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:12.969252 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:12.970808 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:12.971263 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:12.971369 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:12.974813 (+ 3444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:12.974818 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:12.974822 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:12.974860 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:12.978099 (+ 3239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:12.979238 (+ 1139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:12.979244 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:12.979248 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:12.979492 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:12.980125 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:12.980130 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:12.980133 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:12.980446 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:12.980570 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:12.981292 (+ 722us) server_negotiation.cc:299] Negotiation successful
0928 14:56:12.981458 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":275,"threads_started":1}
I20260928 14:56:12.982736 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:12.963055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43337 (local address 127.31.40.3:52427)
0928 14:56:12.963959 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:12.964002 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:12.965925 (+ 1923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:12.966722 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:12.966731 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:12.967419 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:56:12.968102 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:12.968123 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:12.969413 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:12.969417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:12.970661 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:12.970673 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:12.971461 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:12.971465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:12.971670 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:12.972417 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:56:12.972439 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:56:12.974596 (+ 2157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:56:12.978248 (+ 3652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:12.978257 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:12.978278 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:12.978880 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:12.979643 (+ 763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:12.979647 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:12.979649 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:12.979939 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:12.980567 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:56:12.980576 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:56:12.980918 (+ 342us) client_negotiation.cc:769] Sending connection context
0928 14:56:12.981197 (+ 279us) client_negotiation.cc:240] Negotiation successful
0928 14:56:12.981596 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":382,"threads_started":1}
I20260928 14:56:12.984206 479 heartbeater.cc:347] Connected to a master server at 127.31.40.62:43337
I20260928 14:56:12.984512 479 heartbeater.cc:464] Registering TS with master...
I20260928 14:56:12.985239 479 heartbeater.cc:511] Master 127.31.40.62:43337 requested a full tablet report, sending...
I20260928 14:56:12.986651 32484 ts_manager.cc:194] Registered new tserver with Master: e6f6dc3b2d2b46c4b068562b1568c2db (127.31.40.3:33741)
I20260928 14:56:12.987419 32484 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.3@KRBTEST.COM'} at 127.31.40.3:52427
I20260928 14:56:12.992477 31904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:56:13.011577 481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:12.998807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41498 (local address 127.31.40.62:43337)
0928 14:56:12.998964 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:12.998968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:12.999457 (+ 489us) server_negotiation.cc:407] Connection header received
0928 14:56:12.999623 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:12.999627 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:12.999701 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:12.999783 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:13.001299 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:13.002088 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:13.003460 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:13.003846 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:13.004003 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:13.006317 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:13.006322 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:13.006325 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:13.006356 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:13.008997 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:13.009664 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:13.009667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:13.009670 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:13.009805 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:13.010237 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:13.010240 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:13.010242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:13.010559 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:13.010732 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:13.011037 (+ 305us) server_negotiation.cc:299] Negotiation successful
0928 14:56:13.011206 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 14:56:13.018324 32484 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41498:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:56:13.026477 32484 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:56:13.052839 492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:13.040088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:41391 (local address 127.0.0.1:38872)
0928 14:56:13.041172 (+ 1084us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:13.041210 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:13.041410 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:13.042512 (+ 1102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:13.042519 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:13.042555 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:56:13.042894 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:13.042907 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:13.045553 (+ 2646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:13.045563 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:13.048844 (+ 3281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:13.048868 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:13.051599 (+ 2731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:13.051606 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:13.051828 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:13.051947 (+ 119us) client_negotiation.cc:769] Sending connection context
0928 14:56:13.052114 (+ 167us) client_negotiation.cc:240] Negotiation successful
0928 14:56:13.052362 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":895,"mutex_wait_us":51,"thread_start_us":772,"threads_started":1}
I20260928 14:56:13.052889 493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:13.040192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38872 (local address 127.31.40.1:41391)
0928 14:56:13.040802 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:13.040814 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:13.041322 (+ 508us) server_negotiation.cc:407] Connection header received
0928 14:56:13.041626 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:13.041636 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:13.041928 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:13.042166 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:56:13.043118 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:13.045323 (+ 2205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:13.049115 (+ 3792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:13.051082 (+ 1967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:13.051205 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:13.051462 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:13.052386 (+ 924us) server_negotiation.cc:299] Negotiation successful
0928 14:56:13.052579 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":342,"threads_started":1}
I20260928 14:56:13.053453 495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:13.041842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46542 (local address 127.31.40.2:38065)
0928 14:56:13.042827 (+ 985us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:13.042837 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:13.042867 (+ 30us) server_negotiation.cc:407] Connection header received
0928 14:56:13.043002 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:13.043013 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:13.043295 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:13.044652 (+ 1357us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:56:13.044948 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:13.047175 (+ 2227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:13.050397 (+ 3222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:13.052021 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:13.052181 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:13.052393 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:13.052954 (+ 561us) server_negotiation.cc:299] Negotiation successful
0928 14:56:13.053131 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":819,"thread_start_us":667,"threads_started":1}
I20260928 14:56:13.053804 494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:13.041583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.2:38065 (local address 127.0.0.1:46542)
0928 14:56:13.042152 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:13.042170 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:13.042286 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:13.043619 (+ 1333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:13.043627 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:13.043661 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:56:13.044364 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:13.044382 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:13.047397 (+ 3015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:13.047403 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:13.050224 (+ 2821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:13.050238 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:13.052262 (+ 2024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:13.052268 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:13.052520 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:13.052561 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 14:56:13.052638 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 14:56:13.052789 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"mutex_wait_us":1,"thread_start_us":304,"threads_started":1}
I20260928 14:56:13.057570 32745 tablet_service.cc:1467] Processing CreateTablet for tablet 890c4b4b1d184e33b50e7f2310346eea (DEFAULT_TABLE table=test-table [id=b91b8db6af3d4a8aa9ba494382921dd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:56:13.057435 32610 tablet_service.cc:1467] Processing CreateTablet for tablet 890c4b4b1d184e33b50e7f2310346eea (DEFAULT_TABLE table=test-table [id=b91b8db6af3d4a8aa9ba494382921dd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:56:13.058892 32610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 890c4b4b1d184e33b50e7f2310346eea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:56:13.058912 32745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 890c4b4b1d184e33b50e7f2310346eea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:56:13.059187 497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:13.042917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.3:33741 (local address 127.0.0.1:53416)
0928 14:56:13.050803 (+ 7886us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:13.050823 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:13.050966 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:13.051796 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:13.051800 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:13.051825 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:56:13.052494 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:13.052511 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:13.054443 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:13.054447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:13.056528 (+ 2081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:13.056545 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:13.058286 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:13.058293 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:13.058467 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:13.058507 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 14:56:13.058644 (+ 137us) client_negotiation.cc:240] Negotiation successful
0928 14:56:13.058781 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7749,"mutex_wait_us":177,"thread_start_us":7600,"threads_started":1}
I20260928 14:56:13.060138 496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:13.042551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53416 (local address 127.31.40.3:33741)
0928 14:56:13.049457 (+ 6906us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:13.049467 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:13.051094 (+ 1627us) server_negotiation.cc:407] Connection header received
0928 14:56:13.051203 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:13.051212 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:13.051508 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:13.051719 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:56:13.052706 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:13.054285 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:13.056702 (+ 2417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:13.058079 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:13.058213 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:13.058434 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:13.059594 (+ 1160us) server_negotiation.cc:299] Negotiation successful
0928 14:56:13.059748 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6711,"thread_start_us":6553,"threads_started":1}
I20260928 14:56:13.064090 413 tablet_service.cc:1467] Processing CreateTablet for tablet 890c4b4b1d184e33b50e7f2310346eea (DEFAULT_TABLE table=test-table [id=b91b8db6af3d4a8aa9ba494382921dd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:56:13.065608 413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 890c4b4b1d184e33b50e7f2310346eea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:56:13.071976 498 tablet_bootstrap.cc:492] T 890c4b4b1d184e33b50e7f2310346eea P 8f56ad9a62474242bb63993a297328d6: Bootstrap starting.
I20260928 14:56:13.076313 498 tablet_bootstrap.cc:654] T 890c4b4b1d184e33b50e7f2310346eea P 8f56ad9a62474242bb63993a297328d6: Neither blocks nor log segments found. Creating new log.
I20260928 14:56:13.076742 499 tablet_bootstrap.cc:492] T 890c4b4b1d184e33b50e7f2310346eea P 2c6a384011f1414f9fc0933fe3fc66c6: Bootstrap starting.
I20260928 14:56:13.077793 498 log.cc:824] T 890c4b4b1d184e33b50e7f2310346eea P 8f56ad9a62474242bb63993a297328d6: Log is configured to *not* fsync() on all Append() calls
I20260928 14:56:13.080338 499 tablet_bootstrap.cc:654] T 890c4b4b1d184e33b50e7f2310346eea P 2c6a384011f1414f9fc0933fe3fc66c6: Neither blocks nor log segments found. Creating new log.
I20260928 14:56:13.081317 498 tablet_bootstrap.cc:492] T 890c4b4b1d184e33b50e7f2310346eea P 8f56ad9a62474242bb63993a297328d6: No bootstrap required, opened a new log
I20260928 14:56:13.081703 498 ts_tablet_manager.cc:1397] T 890c4b4b1d184e33b50e7f2310346eea P 8f56ad9a62474242bb63993a297328d6: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260928 14:56:13.082093 499 log.cc:824] T 890c4b4b1d184e33b50e7f2310346eea P 2c6a384011f1414f9fc0933fe3fc66c6: Log is configured to *not* fsync() on all Append() calls
I20260928 14:56:13.082376 501 tablet_bootstrap.cc:492] T 890c4b4b1d184e33b50e7f2310346eea P e6f6dc3b2d2b46c4b068562b1568c2db: Bootstrap starting.
I20260928 14:56:13.085338 501 tablet_bootstrap.cc:654] T 890c4b4b1d184e33b50e7f2310346eea P e6f6dc3b2d2b46c4b068562b1568c2db: Neither blocks nor log segments found. Creating new log.
I20260928 14:56:13.086304 501 log.cc:824] T 890c4b4b1d184e33b50e7f2310346eea P e6f6dc3b2d2b46c4b068562b1568c2db: Log is configured to *not* fsync() on all Append() calls
I20260928 14:56:13.086527 499 tablet_bootstrap.cc:492] T 890c4b4b1d184e33b50e7f2310346eea P 2c6a384011f1414f9fc0933fe3fc66c6: No bootstrap required, opened a new log
I20260928 14:56:13.086805 499 ts_tablet_manager.cc:1397] T 890c4b4b1d184e33b50e7f2310346eea P 2c6a384011f1414f9fc0933fe3fc66c6: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.002s
I20260928 14:56:13.088873 501 tablet_bootstrap.cc:492] T 890c4b4b1d184e33b50e7f2310346eea P e6f6dc3b2d2b46c4b068562b1568c2db: No bootstrap required, opened a new log
I20260928 14:56:13.089136 501 ts_tablet_manager.cc:1397] T 890c4b4b1d184e33b50e7f2310346eea P e6f6dc3b2d2b46c4b068562b1568c2db: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260928 14:56:13.093012 498 raft_consensus.cc:348] T 890c4b4b1d184e33b50e7f2310346eea P 8f56ad9a62474242bb63993a297328d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c6a384011f1414f9fc0933fe3fc66c6" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 41391 } } peers { permanent_uuid: "8f56ad9a62474242bb63993a297328d6" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 38065 } } peers { permanent_uuid: "e6f6dc3b2d2b46c4b068562b1568c2db" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33741 } }
I20260928 14:56:13.093469 498 raft_consensus.cc:374] T 890c4b4b1d184e33b50e7f2310346eea P 8f56ad9a62474242bb63993a297328d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:56:13.093617 498 raft_consensus.cc:729] T 890c4b4b1d184e33b50e7f2310346eea P 8f56ad9a62474242bb63993a297328d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f56ad9a62474242bb63993a297328d6, State: Initialized, Role: FOLLOWER
I20260928 14:56:13.094324 498 consensus_queue.cc:260] T 890c4b4b1d184e33b50e7f2310346eea P 8f56ad9a62474242bb63993a297328d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c6a384011f1414f9fc0933fe3fc66c6" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 41391 } } peers { permanent_uuid: "8f56ad9a62474242bb63993a297328d6" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 38065 } } peers { permanent_uuid: "e6f6dc3b2d2b46c4b068562b1568c2db" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33741 } }
I20260928 14:56:13.096700 342 heartbeater.cc:503] Master 127.31.40.62:43337 was elected leader, sending a full tablet report...
I20260928 14:56:13.097621 499 raft_consensus.cc:348] T 890c4b4b1d184e33b50e7f2310346eea P 2c6a384011f1414f9fc0933fe3fc66c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c6a384011f1414f9fc0933fe3fc66c6" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 41391 } } peers { permanent_uuid: "8f56ad9a62474242bb63993a297328d6" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 38065 } } peers { permanent_uuid: "e6f6dc3b2d2b46c4b068562b1568c2db" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33741 } }
I20260928 14:56:13.098111 499 raft_consensus.cc:374] T 890c4b4b1d184e33b50e7f2310346eea P 2c6a384011f1414f9fc0933fe3fc66c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:56:13.098260 499 raft_consensus.cc:729] T 890c4b4b1d184e33b50e7f2310346eea P 2c6a384011f1414f9fc0933fe3fc66c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c6a384011f1414f9fc0933fe3fc66c6, State: Initialized, Role: FOLLOWER
I20260928 14:56:13.098827 498 ts_tablet_manager.cc:1428] T 890c4b4b1d184e33b50e7f2310346eea P 8f56ad9a62474242bb63993a297328d6: Time spent starting tablet: real 0.017s user 0.005s sys 0.008s
I20260928 14:56:13.098932 499 consensus_queue.cc:260] T 890c4b4b1d184e33b50e7f2310346eea P 2c6a384011f1414f9fc0933fe3fc66c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c6a384011f1414f9fc0933fe3fc66c6" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 41391 } } peers { permanent_uuid: "8f56ad9a62474242bb63993a297328d6" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 38065 } } peers { permanent_uuid: "e6f6dc3b2d2b46c4b068562b1568c2db" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33741 } }
I20260928 14:56:13.098178 501 raft_consensus.cc:348] T 890c4b4b1d184e33b50e7f2310346eea P e6f6dc3b2d2b46c4b068562b1568c2db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c6a384011f1414f9fc0933fe3fc66c6" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 41391 } } peers { permanent_uuid: "8f56ad9a62474242bb63993a297328d6" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 38065 } } peers { permanent_uuid: "e6f6dc3b2d2b46c4b068562b1568c2db" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33741 } }
I20260928 14:56:13.100157 501 raft_consensus.cc:374] T 890c4b4b1d184e33b50e7f2310346eea P e6f6dc3b2d2b46c4b068562b1568c2db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:56:13.100284 501 raft_consensus.cc:729] T 890c4b4b1d184e33b50e7f2310346eea P e6f6dc3b2d2b46c4b068562b1568c2db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6f6dc3b2d2b46c4b068562b1568c2db, State: Initialized, Role: FOLLOWER
I20260928 14:56:13.100661 499 ts_tablet_manager.cc:1428] T 890c4b4b1d184e33b50e7f2310346eea P 2c6a384011f1414f9fc0933fe3fc66c6: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260928 14:56:13.100879 501 consensus_queue.cc:260] T 890c4b4b1d184e33b50e7f2310346eea P e6f6dc3b2d2b46c4b068562b1568c2db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c6a384011f1414f9fc0933fe3fc66c6" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 41391 } } peers { permanent_uuid: "8f56ad9a62474242bb63993a297328d6" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 38065 } } peers { permanent_uuid: "e6f6dc3b2d2b46c4b068562b1568c2db" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33741 } }
I20260928 14:56:13.102005 479 heartbeater.cc:503] Master 127.31.40.62:43337 was elected leader, sending a full tablet report...
I20260928 14:56:13.102692 501 ts_tablet_manager.cc:1428] T 890c4b4b1d184e33b50e7f2310346eea P e6f6dc3b2d2b46c4b068562b1568c2db: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
W20260928 14:56:13.147482 32676 tablet.cc:2369] T 890c4b4b1d184e33b50e7f2310346eea P 2c6a384011f1414f9fc0933fe3fc66c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:56:13.156821 505 raft_consensus.cc:482] T 890c4b4b1d184e33b50e7f2310346eea P 2c6a384011f1414f9fc0933fe3fc66c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:56:13.157162 505 raft_consensus.cc:504] T 890c4b4b1d184e33b50e7f2310346eea P 2c6a384011f1414f9fc0933fe3fc66c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c6a384011f1414f9fc0933fe3fc66c6" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 41391 } } peers { permanent_uuid: "8f56ad9a62474242bb63993a297328d6" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 38065 } } peers { permanent_uuid: "e6f6dc3b2d2b46c4b068562b1568c2db" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33741 } }
I20260928 14:56:13.159045 505 leader_election.cc:288] T 890c4b4b1d184e33b50e7f2310346eea P 2c6a384011f1414f9fc0933fe3fc66c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f56ad9a62474242bb63993a297328d6 (127.31.40.2:38065), e6f6dc3b2d2b46c4b068562b1568c2db (127.31.40.3:33741)
I20260928 14:56:13.167491 496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:13.160805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:38385 (local address 127.31.40.3:33741)
0928 14:56:13.161020 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:13.161025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:13.161262 (+ 237us) server_negotiation.cc:407] Connection header received
0928 14:56:13.161468 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:13.161473 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:13.161542 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:13.161645 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:56:13.162168 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:13.162906 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:13.164985 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:13.166015 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:13.166211 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:13.166387 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:13.166891 (+ 504us) server_negotiation.cc:299] Negotiation successful
0928 14:56:13.167027 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 14:56:13.167706 495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:13.159599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:48095 (local address 127.31.40.2:38065)
0928 14:56:13.159900 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:13.159911 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:13.160473 (+ 562us) server_negotiation.cc:407] Connection header received
0928 14:56:13.160759 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:13.160770 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:13.160875 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:13.160985 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:56:13.161620 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:13.162458 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:13.164988 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:13.166203 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:13.166430 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:13.166560 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:13.167081 (+ 521us) server_negotiation.cc:299] Negotiation successful
0928 14:56:13.167226 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 14:56:13.167728 508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:13.160498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.3:33741 (local address 127.31.40.1:38385)
0928 14:56:13.161159 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:13.161176 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:13.161333 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:13.161662 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:13.161666 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:13.161689 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:56:13.162017 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:13.162026 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:13.163064 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:13.163069 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:13.164827 (+ 1758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:13.164841 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:13.166379 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:13.166386 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:13.166687 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:13.166714 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 14:56:13.166849 (+ 135us) client_negotiation.cc:240] Negotiation successful
0928 14:56:13.166993 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":304,"threads_started":1}
I20260928 14:56:13.168098 507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:13.159543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.2:38065 (local address 127.31.40.1:48095)
0928 14:56:13.160350 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:13.160367 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:13.160559 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:13.161061 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:13.161066 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:13.161095 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:56:13.161470 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:13.161479 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:13.162606 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:13.162615 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:13.164749 (+ 2134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:13.164763 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:13.166709 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:13.166717 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:13.166897 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:13.166924 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 14:56:13.167048 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 14:56:13.167196 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"mutex_wait_us":3,"thread_start_us":475,"threads_started":1}
I20260928 14:56:13.169487 32765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "890c4b4b1d184e33b50e7f2310346eea" candidate_uuid: "2c6a384011f1414f9fc0933fe3fc66c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f56ad9a62474242bb63993a297328d6" is_pre_election: true
I20260928 14:56:13.169544 433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "890c4b4b1d184e33b50e7f2310346eea" candidate_uuid: "2c6a384011f1414f9fc0933fe3fc66c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6f6dc3b2d2b46c4b068562b1568c2db" is_pre_election: true
I20260928 14:56:13.170217 32765 raft_consensus.cc:2454] T 890c4b4b1d184e33b50e7f2310346eea P 8f56ad9a62474242bb63993a297328d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c6a384011f1414f9fc0933fe3fc66c6 in term 0.
I20260928 14:56:13.170267 433 raft_consensus.cc:2454] T 890c4b4b1d184e33b50e7f2310346eea P e6f6dc3b2d2b46c4b068562b1568c2db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c6a384011f1414f9fc0933fe3fc66c6 in term 0.
I20260928 14:56:13.171182 32562 leader_election.cc:302] T 890c4b4b1d184e33b50e7f2310346eea P 2c6a384011f1414f9fc0933fe3fc66c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c6a384011f1414f9fc0933fe3fc66c6, e6f6dc3b2d2b46c4b068562b1568c2db; no voters:
I20260928 14:56:13.171692 505 raft_consensus.cc:2781] T 890c4b4b1d184e33b50e7f2310346eea P 2c6a384011f1414f9fc0933fe3fc66c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:56:13.171949 505 raft_consensus.cc:482] T 890c4b4b1d184e33b50e7f2310346eea P 2c6a384011f1414f9fc0933fe3fc66c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:56:13.172154 505 raft_consensus.cc:3037] T 890c4b4b1d184e33b50e7f2310346eea P 2c6a384011f1414f9fc0933fe3fc66c6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:56:13.174873 505 raft_consensus.cc:504] T 890c4b4b1d184e33b50e7f2310346eea P 2c6a384011f1414f9fc0933fe3fc66c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c6a384011f1414f9fc0933fe3fc66c6" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 41391 } } peers { permanent_uuid: "8f56ad9a62474242bb63993a297328d6" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 38065 } } peers { permanent_uuid: "e6f6dc3b2d2b46c4b068562b1568c2db" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33741 } }
I20260928 14:56:13.175530 505 leader_election.cc:288] T 890c4b4b1d184e33b50e7f2310346eea P 2c6a384011f1414f9fc0933fe3fc66c6 [CANDIDATE]: Term 1 election: Requested vote from peers 8f56ad9a62474242bb63993a297328d6 (127.31.40.2:38065), e6f6dc3b2d2b46c4b068562b1568c2db (127.31.40.3:33741)
I20260928 14:56:13.176056 32765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "890c4b4b1d184e33b50e7f2310346eea" candidate_uuid: "2c6a384011f1414f9fc0933fe3fc66c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f56ad9a62474242bb63993a297328d6"
I20260928 14:56:13.176136 433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "890c4b4b1d184e33b50e7f2310346eea" candidate_uuid: "2c6a384011f1414f9fc0933fe3fc66c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6f6dc3b2d2b46c4b068562b1568c2db"
I20260928 14:56:13.176278 32765 raft_consensus.cc:3037] T 890c4b4b1d184e33b50e7f2310346eea P 8f56ad9a62474242bb63993a297328d6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:56:13.176365 433 raft_consensus.cc:3037] T 890c4b4b1d184e33b50e7f2310346eea P e6f6dc3b2d2b46c4b068562b1568c2db [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:56:13.179106 32765 raft_consensus.cc:2454] T 890c4b4b1d184e33b50e7f2310346eea P 8f56ad9a62474242bb63993a297328d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c6a384011f1414f9fc0933fe3fc66c6 in term 1.
I20260928 14:56:13.179538 32562 leader_election.cc:302] T 890c4b4b1d184e33b50e7f2310346eea P 2c6a384011f1414f9fc0933fe3fc66c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c6a384011f1414f9fc0933fe3fc66c6, 8f56ad9a62474242bb63993a297328d6; no voters:
I20260928 14:56:13.179854 505 raft_consensus.cc:2781] T 890c4b4b1d184e33b50e7f2310346eea P 2c6a384011f1414f9fc0933fe3fc66c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:56:13.179919 433 raft_consensus.cc:2454] T 890c4b4b1d184e33b50e7f2310346eea P e6f6dc3b2d2b46c4b068562b1568c2db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c6a384011f1414f9fc0933fe3fc66c6 in term 1.
I20260928 14:56:13.180692 505 raft_consensus.cc:686] T 890c4b4b1d184e33b50e7f2310346eea P 2c6a384011f1414f9fc0933fe3fc66c6 [term 1 LEADER]: Becoming Leader. State: Replica: 2c6a384011f1414f9fc0933fe3fc66c6, State: Running, Role: LEADER
I20260928 14:56:13.181241 505 consensus_queue.cc:237] T 890c4b4b1d184e33b50e7f2310346eea P 2c6a384011f1414f9fc0933fe3fc66c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c6a384011f1414f9fc0933fe3fc66c6" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 41391 } } peers { permanent_uuid: "8f56ad9a62474242bb63993a297328d6" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 38065 } } peers { permanent_uuid: "e6f6dc3b2d2b46c4b068562b1568c2db" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33741 } }
I20260928 14:56:13.186780 32482 catalog_manager.cc:5495] T 890c4b4b1d184e33b50e7f2310346eea P 2c6a384011f1414f9fc0933fe3fc66c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c6a384011f1414f9fc0933fe3fc66c6 (127.31.40.1). New cstate: current_term: 1 leader_uuid: "2c6a384011f1414f9fc0933fe3fc66c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c6a384011f1414f9fc0933fe3fc66c6" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 41391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f56ad9a62474242bb63993a297328d6" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 38065 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6f6dc3b2d2b46c4b068562b1568c2db" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33741 } health_report { overall_health: UNKNOWN } } }
W20260928 14:56:13.187933 343 tablet.cc:2369] T 890c4b4b1d184e33b50e7f2310346eea P 8f56ad9a62474242bb63993a297328d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:56:13.211611 480 tablet.cc:2369] T 890c4b4b1d184e33b50e7f2310346eea P e6f6dc3b2d2b46c4b068562b1568c2db: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 14:56:13 dist-test-slave-j968 krb5kdc[32439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.1@KRBTEST.COM
I20260928 14:56:13.226372 493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:13.210337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38876 (local address 127.31.40.1:41391)
0928 14:56:13.210659 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:13.210671 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:13.211297 (+ 626us) server_negotiation.cc:407] Connection header received
0928 14:56:13.211382 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:13.211388 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:13.211488 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:13.211617 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:13.213231 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:13.214091 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:13.215305 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:13.215739 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:13.215855 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:13.220089 (+ 4234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:13.220104 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:13.220122 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:13.220164 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:13.223211 (+ 3047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:13.224127 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:13.224134 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:13.224137 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:13.224285 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:13.224747 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:13.224752 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:13.224754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:13.225176 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:13.225404 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:13.225725 (+ 321us) server_negotiation.cc:299] Negotiation successful
0928 14:56:13.225988 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
Sep 28 14:56:13 dist-test-slave-j968 krb5kdc[32439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607373, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 14:56:13 dist-test-slave-j968 krb5kdc[32439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607373, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.40.1@KRBTEST.COM
I20260928 14:56:13.274405 493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:13.259744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38892 (local address 127.31.40.1:41391)
0928 14:56:13.260007 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:13.260015 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:13.260374 (+ 359us) server_negotiation.cc:407] Connection header received
0928 14:56:13.260601 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:13.260608 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:13.260686 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:13.260796 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:13.262244 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:13.263014 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:13.264355 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:13.264767 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:13.264905 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:13.268923 (+ 4018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:13.268928 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:13.268932 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:13.268971 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:13.271644 (+ 2673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:13.272540 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:13.272544 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:13.272546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:13.272688 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:13.273153 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:13.273157 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:13.273158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:13.273431 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:13.273713 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:13.273957 (+ 244us) server_negotiation.cc:299] Negotiation successful
0928 14:56:13.274141 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Sep 28 14:56:13 dist-test-slave-j968 krb5kdc[32439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 14:56:13 dist-test-slave-j968 krb5kdc[32439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.1@KRBTEST.COM
I20260928 14:56:13.330899 493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:13.312882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38908 (local address 127.31.40.1:41391)
0928 14:56:13.313181 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:13.313189 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:13.313520 (+ 331us) server_negotiation.cc:407] Connection header received
0928 14:56:13.313761 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:13.313765 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:13.313862 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:13.313977 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:13.316355 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:13.317248 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:13.318963 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:13.319393 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:13.319520 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:13.324105 (+ 4585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:13.324111 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:13.324115 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:13.324160 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:13.327732 (+ 3572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:13.328760 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:13.328766 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:13.328769 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:13.328998 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:13.329473 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:13.329477 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:13.329479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:13.329788 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:13.329999 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:13.330438 (+ 439us) server_negotiation.cc:299] Negotiation successful
0928 14:56:13.330630 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141}
W20260928 14:56:13.332952 32589 tablet_service.cc:2908] Rejecting scan request for tablet 890c4b4b1d184e33b50e7f2310346eea: Uninitialized: clean time has not yet been initialized
I20260928 14:56:13.336443 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 32546
I20260928 14:56:13.352499 493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:13.337607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38920 (local address 127.31.40.1:41391)
0928 14:56:13.337773 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:13.337778 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:13.338263 (+ 485us) server_negotiation.cc:407] Connection header received
0928 14:56:13.338428 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:13.338432 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:13.338552 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:13.338649 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:13.340041 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:13.340831 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:13.342530 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:13.343357 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:13.343489 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:13.346692 (+ 3203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:13.346697 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:13.346701 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:13.346739 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:13.349632 (+ 2893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:13.350435 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:13.350439 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:13.350441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:13.350564 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:13.351024 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:13.351028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:13.351030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:13.351319 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:13.351516 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:13.352025 (+ 509us) server_negotiation.cc:299] Negotiation successful
0928 14:56:13.352241 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 14:56:13.353214 32670 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:56:13.464020 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 32546
I20260928 14:56:13.481650 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 32681
Sep 28 14:56:13 dist-test-slave-j968 krb5kdc[32439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.2@KRBTEST.COM
I20260928 14:56:13.496829 495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:13.482654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46556 (local address 127.31.40.2:38065)
0928 14:56:13.482841 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:13.482849 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:13.482872 (+ 23us) server_negotiation.cc:407] Connection header received
0928 14:56:13.482942 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:13.482946 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:13.483040 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:13.483157 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:13.484371 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:13.485129 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:13.486384 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:13.486794 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:13.486932 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:13.490938 (+ 4006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:13.490954 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:13.490971 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:13.491009 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:13.493810 (+ 2801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:13.494681 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:13.494689 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:13.494692 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:13.494820 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:13.495343 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:13.495347 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:13.495349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:13.495742 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:13.496054 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:13.496394 (+ 340us) server_negotiation.cc:299] Negotiation successful
0928 14:56:13.496579 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 14:56:13.497579 337 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:56:13.596160 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 32681
I20260928 14:56:13.621388 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 348
Sep 28 14:56:13 dist-test-slave-j968 krb5kdc[32439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.3@KRBTEST.COM
I20260928 14:56:13.640664 496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:13.622322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53430 (local address 127.31.40.3:33741)
0928 14:56:13.622537 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:13.622546 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:13.622571 (+ 25us) server_negotiation.cc:407] Connection header received
0928 14:56:13.622643 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:13.622651 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:13.622740 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:13.622860 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:13.624331 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:13.625574 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:13.627210 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:13.627852 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:13.628026 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:13.632971 (+ 4945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:13.632994 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:13.633020 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:13.633082 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:13.637209 (+ 4127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:13.637973 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:13.637986 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:13.637989 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:13.638168 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:13.638572 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:13.638579 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:13.638583 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:13.639066 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:13.639384 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:13.639627 (+ 243us) server_negotiation.cc:299] Negotiation successful
0928 14:56:13.640125 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 14:56:13.641492 473 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:56:13.730074 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 348
I20260928 14:56:13.747252 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 32452
I20260928 14:56:13.748653 32515 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:56:13.858574 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 32452
2026-09-28T14:56:13Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4343 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:56:13 dist-test-slave-j968 krb5kdc[528](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:56:13 dist-test-slave-j968 krb5kdc[528](info): set up 2 sockets
Sep 28 14:56:13 dist-test-slave-j968 krb5kdc[528](info): commencing operation
krb5kdc: starting...
W20260928 14:56:15.005553 31904 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.003s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Sep 28 14:56:15 dist-test-slave-j968 krb5kdc[528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:56:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:56:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab.
Entry for principal kudu/127.31.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab.
I20260928 14:56:15.200495 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:34279
--webserver_interface=127.31.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:46027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.40.62:34279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:56:15.535969 541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:56:15.536283 541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:56:15.536458 541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:56:15.544034 541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:56:15.544158 541 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 14:56:15.544235 541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:56:15.544340 541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:56:15.544414 541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:56:15.544521 541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:56:15.554747 541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:46027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.40.62:34279
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:34279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:56:15.556210 541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:56:15.557710 541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:56:15.570623 546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:15.570677 547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:15.570746 549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:15.571676 541 server_base.cc:1034] running on GCE node
I20260928 14:56:15.572605 541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:56:15.574164 541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:56:15.575429 541 hybrid_clock.cc:648] HybridClock initialized: now 1790607375575366 us; error 45 us; skew 500 ppm
I20260928 14:56:15.575909 541 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:56:15 dist-test-slave-j968 krb5kdc[528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607375, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:56:15.589242 541 init.cc:377] Logged in from keytab as kudu/127.31.40.62@KRBTEST.COM (short username kudu)
I20260928 14:56:15.591028 541 webserver.cc:460] Webserver started at http://127.31.40.62:34687/ using document root <none> and password file <none>
I20260928 14:56:15.591830 541 fs_manager.cc:359] Metadata directory not provided
I20260928 14:56:15.592015 541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:56:15.592366 541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:56:15.595468 541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/master-0/data/instance:
uuid: "633f7fb706e243119cb41b0ad22c5be6"
format_stamp: "Formatted at 2026-09-28 14:56:15 on dist-test-slave-j968"
server_key: "ac63f248ba98c8a3c637668fe23ed6c6"
server_key_iv: "f99e7193349a784f0c8420ebb3560d8e"
server_key_version: "encryptionkey@0"
I20260928 14:56:15.596308 541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/master-0/wal/instance:
uuid: "633f7fb706e243119cb41b0ad22c5be6"
format_stamp: "Formatted at 2026-09-28 14:56:15 on dist-test-slave-j968"
server_key: "ac63f248ba98c8a3c637668fe23ed6c6"
server_key_iv: "f99e7193349a784f0c8420ebb3560d8e"
server_key_version: "encryptionkey@0"
I20260928 14:56:15.601828 541 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 14:56:15.605614 557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:15.607019 541 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260928 14:56:15.607262 541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/master-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/master-0/wal
uuid: "633f7fb706e243119cb41b0ad22c5be6"
format_stamp: "Formatted at 2026-09-28 14:56:15 on dist-test-slave-j968"
server_key: "ac63f248ba98c8a3c637668fe23ed6c6"
server_key_iv: "f99e7193349a784f0c8420ebb3560d8e"
server_key_version: "encryptionkey@0"
I20260928 14:56:15.607496 541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:56:15.630494 541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:56:15.634927 541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:56:15.635236 541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:56:15.657255 541 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.62:34279
I20260928 14:56:15.657262 618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.62:34279 every 8 connection(s)
I20260928 14:56:15.658713 541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
I20260928 14:56:15.659361 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 541
I20260928 14:56:15.659626 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/master-0/wal/instance
I20260928 14:56:15.660415 31904 external_mini_cluster.cc:1428] Setting key 8649d86290b2e289ec1d4ca5c814fcec
I20260928 14:56:15.663522 619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 14:56:15 dist-test-slave-j968 krb5kdc[528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:56:15.676851 619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 633f7fb706e243119cb41b0ad22c5be6: Bootstrap starting.
I20260928 14:56:15.680285 619 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 633f7fb706e243119cb41b0ad22c5be6: Neither blocks nor log segments found. Creating new log.
I20260928 14:56:15.681463 619 log.cc:824] T 00000000000000000000000000000000 P 633f7fb706e243119cb41b0ad22c5be6: Log is configured to *not* fsync() on all Append() calls
I20260928 14:56:15.684938 619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 633f7fb706e243119cb41b0ad22c5be6: No bootstrap required, opened a new log
I20260928 14:56:15.685254 622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:15.662580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57170 (local address 127.31.40.62:34279)
0928 14:56:15.663382 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:15.663397 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:15.663450 (+ 53us) server_negotiation.cc:407] Connection header received
0928 14:56:15.665865 (+ 2415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:15.665898 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:15.666429 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:15.666919 (+ 490us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:15.668375 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:15.670510 (+ 2135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:15.671813 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:15.672875 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:15.673063 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:15.677269 (+ 4206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:15.677277 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:15.677293 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:15.677337 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:15.680464 (+ 3127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:15.681312 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:15.681321 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:15.681329 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:15.681543 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:15.682040 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:15.682052 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:15.682058 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:15.682695 (+ 637us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:15.683012 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:15.683411 (+ 399us) server_negotiation.cc:299] Negotiation successful
0928 14:56:15.683851 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":344,"threads_started":1}
I20260928 14:56:15.693727 619 raft_consensus.cc:348] T 00000000000000000000000000000000 P 633f7fb706e243119cb41b0ad22c5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "633f7fb706e243119cb41b0ad22c5be6" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34279 } }
I20260928 14:56:15.694089 619 raft_consensus.cc:374] T 00000000000000000000000000000000 P 633f7fb706e243119cb41b0ad22c5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:56:15.694201 619 raft_consensus.cc:729] T 00000000000000000000000000000000 P 633f7fb706e243119cb41b0ad22c5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 633f7fb706e243119cb41b0ad22c5be6, State: Initialized, Role: FOLLOWER
I20260928 14:56:15.694752 619 consensus_queue.cc:260] T 00000000000000000000000000000000 P 633f7fb706e243119cb41b0ad22c5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "633f7fb706e243119cb41b0ad22c5be6" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34279 } }
I20260928 14:56:15.694950 619 raft_consensus.cc:388] T 00000000000000000000000000000000 P 633f7fb706e243119cb41b0ad22c5be6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:56:15.695042 619 raft_consensus.cc:482] T 00000000000000000000000000000000 P 633f7fb706e243119cb41b0ad22c5be6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:56:15.695173 619 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 633f7fb706e243119cb41b0ad22c5be6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:56:15.697840 619 raft_consensus.cc:504] T 00000000000000000000000000000000 P 633f7fb706e243119cb41b0ad22c5be6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "633f7fb706e243119cb41b0ad22c5be6" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34279 } }
I20260928 14:56:15.698338 619 leader_election.cc:302] T 00000000000000000000000000000000 P 633f7fb706e243119cb41b0ad22c5be6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 633f7fb706e243119cb41b0ad22c5be6; no voters:
I20260928 14:56:15.698907 619 leader_election.cc:288] T 00000000000000000000000000000000 P 633f7fb706e243119cb41b0ad22c5be6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:56:15.699107 624 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 633f7fb706e243119cb41b0ad22c5be6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:56:15.699819 624 raft_consensus.cc:686] T 00000000000000000000000000000000 P 633f7fb706e243119cb41b0ad22c5be6 [term 1 LEADER]: Becoming Leader. State: Replica: 633f7fb706e243119cb41b0ad22c5be6, State: Running, Role: LEADER
I20260928 14:56:15.700393 624 consensus_queue.cc:237] T 00000000000000000000000000000000 P 633f7fb706e243119cb41b0ad22c5be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "633f7fb706e243119cb41b0ad22c5be6" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34279 } }
I20260928 14:56:15.701113 619 sys_catalog.cc:564] T 00000000000000000000000000000000 P 633f7fb706e243119cb41b0ad22c5be6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:56:15.703147 625 sys_catalog.cc:455] T 00000000000000000000000000000000 P 633f7fb706e243119cb41b0ad22c5be6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "633f7fb706e243119cb41b0ad22c5be6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "633f7fb706e243119cb41b0ad22c5be6" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34279 } } }
I20260928 14:56:15.703552 625 sys_catalog.cc:458] T 00000000000000000000000000000000 P 633f7fb706e243119cb41b0ad22c5be6 [sys.catalog]: This master's current role is: LEADER
I20260928 14:56:15.703675 626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 633f7fb706e243119cb41b0ad22c5be6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 633f7fb706e243119cb41b0ad22c5be6. Latest consensus state: current_term: 1 leader_uuid: "633f7fb706e243119cb41b0ad22c5be6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "633f7fb706e243119cb41b0ad22c5be6" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34279 } } }
I20260928 14:56:15.704034 626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 633f7fb706e243119cb41b0ad22c5be6 [sys.catalog]: This master's current role is: LEADER
I20260928 14:56:15.704830 631 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:56:15.711930 631 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:56:15.721933 631 catalog_manager.cc:1347] Generated new cluster ID: c09ba679f8db4224ac0919f3e12610ee
I20260928 14:56:15.722087 631 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:56:15.743300 631 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:56:15.745486 631 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:56:15.766331 631 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 633f7fb706e243119cb41b0ad22c5be6: Generated new TSK 0
I20260928 14:56:15.767532 631 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.40.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab.
Entry for principal kudu/127.31.40.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab.
I20260928 14:56:15.824720 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.1:0
--local_ip_for_outbound_sockets=127.31.40.1
--webserver_interface=127.31.40.1
--webserver_port=0
--tserver_master_addrs=127.31.40.62:34279
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab
--principal=kudu/127.31.40.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:46027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:56:16.189841 645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:56:16.190176 645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:56:16.190346 645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:56:16.198835 645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:56:16.198967 645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:56:16.199144 645 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:56:16.199258 645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.1
I20260928 14:56:16.209646 645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:46027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab
--principal=kudu/127.31.40.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.40.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.40.62:34279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.1
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:56:16.211129 645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:56:16.212736 645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:56:16.227636 651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:16.227727 653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:16.227677 650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:16.228663 645 server_base.cc:1034] running on GCE node
I20260928 14:56:16.229411 645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:56:16.230463 645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:56:16.231753 645 hybrid_clock.cc:648] HybridClock initialized: now 1790607376231679 us; error 44 us; skew 500 ppm
I20260928 14:56:16.232146 645 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:56:16 dist-test-slave-j968 krb5kdc[528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607376, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:56:16.235579 645 init.cc:377] Logged in from keytab as kudu/127.31.40.1@KRBTEST.COM (short username kudu)
I20260928 14:56:16.237229 645 webserver.cc:460] Webserver started at http://127.31.40.1:45375/ using document root <none> and password file <none>
I20260928 14:56:16.237975 645 fs_manager.cc:359] Metadata directory not provided
I20260928 14:56:16.238107 645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:56:16.238436 645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:56:16.241555 645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/ts-0/data/instance:
uuid: "ac470d791ea04f708719e48abd721021"
format_stamp: "Formatted at 2026-09-28 14:56:16 on dist-test-slave-j968"
server_key: "6afce04742ccbf611bf0038be2ffec5c"
server_key_iv: "2f9b7516b607499ce99c7c14123ca936"
server_key_version: "encryptionkey@0"
I20260928 14:56:16.242419 645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/ts-0/wal/instance:
uuid: "ac470d791ea04f708719e48abd721021"
format_stamp: "Formatted at 2026-09-28 14:56:16 on dist-test-slave-j968"
server_key: "6afce04742ccbf611bf0038be2ffec5c"
server_key_iv: "2f9b7516b607499ce99c7c14123ca936"
server_key_version: "encryptionkey@0"
I20260928 14:56:16.247808 645 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260928 14:56:16.251641 660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:16.253232 645 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:56:16.253450 645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/ts-0/wal
uuid: "ac470d791ea04f708719e48abd721021"
format_stamp: "Formatted at 2026-09-28 14:56:16 on dist-test-slave-j968"
server_key: "6afce04742ccbf611bf0038be2ffec5c"
server_key_iv: "2f9b7516b607499ce99c7c14123ca936"
server_key_version: "encryptionkey@0"
I20260928 14:56:16.253711 645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:56:16.273849 645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:56:16.278463 645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:56:16.278760 645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:56:16.281693 645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:56:16.281806 645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:16.282003 645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:56:16.282084 645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 14:56:16 dist-test-slave-j968 krb5kdc[528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607376, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.1@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:56:16.308048 669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:16.282239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:40409 (local address 127.31.40.62:34279)
0928 14:56:16.282822 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:16.282829 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:16.284062 (+ 1233us) server_negotiation.cc:407] Connection header received
0928 14:56:16.286281 (+ 2219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:16.286287 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:16.286399 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:16.286553 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:16.289089 (+ 2536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:16.290097 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:16.292950 (+ 2853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:16.293392 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:16.293584 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:16.299293 (+ 5709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:16.299304 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:16.299312 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:16.299375 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:16.303651 (+ 4276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:16.304814 (+ 1163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:16.304823 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:16.304828 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:16.305025 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:16.305692 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:16.305698 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:16.305701 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:16.306039 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:16.306206 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:16.307351 (+ 1145us) server_negotiation.cc:299] Negotiation successful
0928 14:56:16.307614 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":281,"threads_started":1}
I20260928 14:56:16.309213 670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:16.282941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34279 (local address 127.31.40.1:40409)
0928 14:56:16.283834 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:16.283909 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:16.285976 (+ 2067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:16.286883 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:16.286894 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:16.287792 (+ 898us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:56:16.288880 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:16.288905 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:16.290274 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:16.290282 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:16.292751 (+ 2469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:16.292766 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:16.293705 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:16.293713 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:16.294121 (+ 408us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:16.295018 (+ 897us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:56:16.295061 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:56:16.298909 (+ 3848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:56:16.303838 (+ 4929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:16.303849 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:16.303902 (+ 53us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:16.304663 (+ 761us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:16.305180 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:16.305185 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:16.305188 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:16.305505 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:16.306293 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:56:16.306304 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:56:16.306826 (+ 522us) client_negotiation.cc:769] Sending connection context
0928 14:56:16.307239 (+ 413us) client_negotiation.cc:240] Negotiation successful
0928 14:56:16.307858 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":351,"threads_started":1}
I20260928 14:56:16.330937 669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:16.316956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:59805 (local address 127.31.40.62:34279)
0928 14:56:16.317201 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:16.317208 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:16.317375 (+ 167us) server_negotiation.cc:407] Connection header received
0928 14:56:16.317610 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:16.317615 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:16.317684 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:16.317785 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:16.319304 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:16.320441 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:16.321767 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:16.322219 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:16.322359 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:16.324430 (+ 2071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:16.324434 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:16.324437 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:16.324468 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:16.327559 (+ 3091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:16.328624 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:16.328632 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:16.328638 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:16.328828 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:16.329233 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:16.329241 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:16.329245 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:16.329677 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:16.329840 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:16.330291 (+ 451us) server_negotiation.cc:299] Negotiation successful
0928 14:56:16.330558 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 14:56:16.331202 745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:16.316497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34279 (local address 127.31.40.1:59805)
0928 14:56:16.317271 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:16.317287 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:16.317461 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:16.317814 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:16.317819 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:16.318613 (+ 794us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:56:16.319134 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:16.319151 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:16.320580 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:16.320590 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:16.321619 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:16.321632 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:16.322404 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:16.322410 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:16.322589 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:16.323243 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:56:16.323262 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:56:16.324192 (+ 930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:56:16.327730 (+ 3538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:16.327734 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:16.327737 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:16.328453 (+ 716us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:16.328961 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:16.328965 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:16.328967 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:16.329102 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:16.329794 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:56:16.329798 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:56:16.330001 (+ 203us) client_negotiation.cc:769] Sending connection context
0928 14:56:16.330189 (+ 188us) client_negotiation.cc:240] Negotiation successful
0928 14:56:16.330601 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":523,"threads_started":1}
I20260928 14:56:16.334817 645 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.1:44729
I20260928 14:56:16.334821 781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.1:44729 every 8 connection(s)
I20260928 14:56:16.336354 645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
I20260928 14:56:16.338665 782 heartbeater.cc:347] Connected to a master server at 127.31.40.62:34279
I20260928 14:56:16.338871 782 heartbeater.cc:464] Registering TS with master...
I20260928 14:56:16.339579 782 heartbeater.cc:511] Master 127.31.40.62:34279 requested a full tablet report, sending...
I20260928 14:56:16.341697 574 ts_manager.cc:194] Registered new tserver with Master: ac470d791ea04f708719e48abd721021 (127.31.40.1:44729)
I20260928 14:56:16.343218 574 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.1@KRBTEST.COM'} at 127.31.40.1:40409
I20260928 14:56:16.345953 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 645
I20260928 14:56:16.346199 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/ts-0/wal/instance
I20260928 14:56:16.346771 31904 external_mini_cluster.cc:1428] Setting key 40d6ca6d68e6954b31da29a1c8d5c676
WARNING: no policy specified for kudu/127.31.40.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab.
Entry for principal kudu/127.31.40.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab.
I20260928 14:56:16.387854 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.2:0
--local_ip_for_outbound_sockets=127.31.40.2
--webserver_interface=127.31.40.2
--webserver_port=0
--tserver_master_addrs=127.31.40.62:34279
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab
--principal=kudu/127.31.40.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:46027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:56:16.741941 786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:56:16.742262 786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:56:16.742424 786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:56:16.750005 786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:56:16.750126 786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:56:16.750296 786 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:56:16.750425 786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.2
I20260928 14:56:16.760849 786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:46027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab
--principal=kudu/127.31.40.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.40.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.40.62:34279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.2
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:56:16.762272 786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:56:16.763672 786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:56:16.778533 791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:16.778558 792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:16.779237 794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:16.781527 786 server_base.cc:1034] running on GCE node
I20260928 14:56:16.782289 786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:56:16.783352 786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:56:16.784606 786 hybrid_clock.cc:648] HybridClock initialized: now 1790607376784531 us; error 52 us; skew 500 ppm
I20260928 14:56:16.784962 786 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:56:16 dist-test-slave-j968 krb5kdc[528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607376, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:56:16.788266 786 init.cc:377] Logged in from keytab as kudu/127.31.40.2@KRBTEST.COM (short username kudu)
I20260928 14:56:16.789870 786 webserver.cc:460] Webserver started at http://127.31.40.2:37365/ using document root <none> and password file <none>
I20260928 14:56:16.790589 786 fs_manager.cc:359] Metadata directory not provided
I20260928 14:56:16.790691 786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:56:16.790966 786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:56:16.794106 786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/ts-1/data/instance:
uuid: "a8d762c4f42841c186f48a56ecc5354f"
format_stamp: "Formatted at 2026-09-28 14:56:16 on dist-test-slave-j968"
server_key: "64eaaf22a857b723fcc770d25887b25e"
server_key_iv: "7bec498151e5f8617c757a55b8f6dde3"
server_key_version: "encryptionkey@0"
I20260928 14:56:16.794883 786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/ts-1/wal/instance:
uuid: "a8d762c4f42841c186f48a56ecc5354f"
format_stamp: "Formatted at 2026-09-28 14:56:16 on dist-test-slave-j968"
server_key: "64eaaf22a857b723fcc770d25887b25e"
server_key_iv: "7bec498151e5f8617c757a55b8f6dde3"
server_key_version: "encryptionkey@0"
I20260928 14:56:16.800463 786 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 14:56:16.804400 801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:16.805968 786 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 14:56:16.806123 786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/ts-1/wal
uuid: "a8d762c4f42841c186f48a56ecc5354f"
format_stamp: "Formatted at 2026-09-28 14:56:16 on dist-test-slave-j968"
server_key: "64eaaf22a857b723fcc770d25887b25e"
server_key_iv: "7bec498151e5f8617c757a55b8f6dde3"
server_key_version: "encryptionkey@0"
I20260928 14:56:16.806394 786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:56:16.827036 786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:56:16.832417 786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:56:16.832747 786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:56:16.835791 786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:56:16.835947 786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 14:56:16.836135 786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:56:16.836215 786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 14:56:16 dist-test-slave-j968 krb5kdc[528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607376, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.2@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:56:16.862818 810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:16.836663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:40285 (local address 127.31.40.62:34279)
0928 14:56:16.837100 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:16.837107 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:16.838862 (+ 1755us) server_negotiation.cc:407] Connection header received
0928 14:56:16.842230 (+ 3368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:16.842239 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:16.842361 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:16.842502 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:16.845446 (+ 2944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:16.846214 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:16.848040 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:16.848718 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:16.848867 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:16.853841 (+ 4974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:16.853851 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:16.853858 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:16.853916 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:16.858152 (+ 4236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:16.859267 (+ 1115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:16.859276 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:16.859283 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:16.859494 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:16.860161 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:16.860166 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:16.860168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:16.860499 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:16.860679 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:16.861857 (+ 1178us) server_negotiation.cc:299] Negotiation successful
0928 14:56:16.862213 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":1358,"threads_started":1}
I20260928 14:56:16.863490 811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:16.837656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34279 (local address 127.31.40.2:40285)
0928 14:56:16.838638 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:16.838698 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:16.841784 (+ 3086us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:16.842836 (+ 1052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:16.842857 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:16.844127 (+ 1270us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:56:16.845220 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:16.845244 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:16.846406 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:16.846410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:16.847789 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:16.847803 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:16.849012 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:16.849020 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:16.849285 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:16.850121 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:56:16.850152 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:56:16.853554 (+ 3402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:56:16.858388 (+ 4834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:16.858397 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:16.858415 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:16.859095 (+ 680us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:16.859645 (+ 550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:16.859649 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:16.859652 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:16.860007 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:16.860650 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:56:16.860662 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:56:16.861284 (+ 622us) client_negotiation.cc:769] Sending connection context
0928 14:56:16.861702 (+ 418us) client_negotiation.cc:240] Negotiation successful
0928 14:56:16.862188 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":384,"threads_started":1}
I20260928 14:56:16.886332 810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:16.870289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:50551 (local address 127.31.40.62:34279)
0928 14:56:16.870552 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:16.870561 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:16.870593 (+ 32us) server_negotiation.cc:407] Connection header received
0928 14:56:16.870807 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:16.870815 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:16.870925 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:16.871065 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:16.872319 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:16.873065 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:16.874789 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:16.875226 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:16.875373 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:16.878013 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:16.878019 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:16.878023 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:16.878068 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:16.882459 (+ 4391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:16.883322 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:16.883331 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:16.883337 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:16.883515 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:16.884115 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:16.884119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:16.884121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:16.884613 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:16.884792 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:16.885766 (+ 974us) server_negotiation.cc:299] Negotiation successful
0928 14:56:16.886018 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 14:56:16.886345 872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:16.869868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34279 (local address 127.31.40.2:50551)
0928 14:56:16.870502 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:16.870524 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:16.870676 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:16.871091 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:16.871094 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:16.871639 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:56:16.872145 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:16.872157 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:16.873219 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:16.873228 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:16.874605 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:16.874621 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:16.875440 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:16.875446 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:16.875641 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:16.876480 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:56:16.876509 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:56:16.877758 (+ 1249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:56:16.882665 (+ 4907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:16.882669 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:16.882672 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:16.883195 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:16.883691 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:16.883699 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:16.883704 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:16.883937 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:16.884870 (+ 933us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:56:16.884877 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:56:16.885124 (+ 247us) client_negotiation.cc:769] Sending connection context
0928 14:56:16.885298 (+ 174us) client_negotiation.cc:240] Negotiation successful
0928 14:56:16.885770 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":482,"threads_started":1}
I20260928 14:56:16.897532 786 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.2:36135
I20260928 14:56:16.897581 922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.2:36135 every 8 connection(s)
I20260928 14:56:16.898787 786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
I20260928 14:56:16.899928 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 786
I20260928 14:56:16.900142 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/ts-1/wal/instance
I20260928 14:56:16.900678 31904 external_mini_cluster.cc:1428] Setting key 4ec08508827d9d09d6ed5af872ad9874
I20260928 14:56:16.901204 923 heartbeater.cc:347] Connected to a master server at 127.31.40.62:34279
I20260928 14:56:16.901499 923 heartbeater.cc:464] Registering TS with master...
I20260928 14:56:16.902315 923 heartbeater.cc:511] Master 127.31.40.62:34279 requested a full tablet report, sending...
I20260928 14:56:16.903846 574 ts_manager.cc:194] Registered new tserver with Master: a8d762c4f42841c186f48a56ecc5354f (127.31.40.2:36135)
I20260928 14:56:16.904728 574 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.2@KRBTEST.COM'} at 127.31.40.2:40285
WARNING: no policy specified for kudu/127.31.40.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab.
Entry for principal kudu/127.31.40.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab.
I20260928 14:56:16.944224 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.3:0
--local_ip_for_outbound_sockets=127.31.40.3
--webserver_interface=127.31.40.3
--webserver_port=0
--tserver_master_addrs=127.31.40.62:34279
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab
--principal=kudu/127.31.40.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:46027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:56:17.288010 927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:56:17.288349 927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:56:17.288511 927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:56:17.295979 927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:56:17.296144 927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:56:17.296301 927 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:56:17.296435 927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.3
I20260928 14:56:17.307014 927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:46027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab
--principal=kudu/127.31.40.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.40.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.40.62:34279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.3
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:56:17.308456 927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:56:17.309959 927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:56:17.324496 935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:17.324564 933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:17.324564 932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:17.325471 927 server_base.cc:1034] running on GCE node
I20260928 14:56:17.326152 927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:56:17.327191 927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:56:17.328492 927 hybrid_clock.cc:648] HybridClock initialized: now 1790607377328425 us; error 37 us; skew 500 ppm
I20260928 14:56:17.328868 927 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:56:17 dist-test-slave-j968 krb5kdc[528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607377, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:56:17.332859 927 init.cc:377] Logged in from keytab as kudu/127.31.40.3@KRBTEST.COM (short username kudu)
I20260928 14:56:17.334815 927 webserver.cc:460] Webserver started at http://127.31.40.3:34817/ using document root <none> and password file <none>
I20260928 14:56:17.335978 927 fs_manager.cc:359] Metadata directory not provided
I20260928 14:56:17.336104 927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:56:17.336490 927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:56:17.340687 927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/ts-2/data/instance:
uuid: "372c38dcee1c43e1af320d7a26511d8c"
format_stamp: "Formatted at 2026-09-28 14:56:17 on dist-test-slave-j968"
server_key: "c21ecac54ad5c6d3110a71abe1e182d0"
server_key_iv: "ad22221fca113d214e5a170b8eb0b8d1"
server_key_version: "encryptionkey@0"
I20260928 14:56:17.341732 927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/ts-2/wal/instance:
uuid: "372c38dcee1c43e1af320d7a26511d8c"
format_stamp: "Formatted at 2026-09-28 14:56:17 on dist-test-slave-j968"
server_key: "c21ecac54ad5c6d3110a71abe1e182d0"
server_key_iv: "ad22221fca113d214e5a170b8eb0b8d1"
server_key_version: "encryptionkey@0"
I20260928 14:56:17.346431 782 heartbeater.cc:503] Master 127.31.40.62:34279 was elected leader, sending a full tablet report...
I20260928 14:56:17.347643 927 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260928 14:56:17.351812 942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:17.353539 927 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 14:56:17.353727 927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/ts-2/wal
uuid: "372c38dcee1c43e1af320d7a26511d8c"
format_stamp: "Formatted at 2026-09-28 14:56:17 on dist-test-slave-j968"
server_key: "c21ecac54ad5c6d3110a71abe1e182d0"
server_key_iv: "ad22221fca113d214e5a170b8eb0b8d1"
server_key_version: "encryptionkey@0"
I20260928 14:56:17.353967 927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:56:17.379079 927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:56:17.383610 927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:56:17.383971 927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:56:17.386687 927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:56:17.386862 927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:17.387034 927 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:56:17.387115 927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 14:56:17 dist-test-slave-j968 krb5kdc[528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607377, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.3@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:56:17.417228 951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:17.387461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:58015 (local address 127.31.40.62:34279)
0928 14:56:17.389844 (+ 2383us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:17.389850 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:17.389873 (+ 23us) server_negotiation.cc:407] Connection header received
0928 14:56:17.391553 (+ 1680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:17.391561 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:17.391680 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:17.391908 (+ 228us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:17.394385 (+ 2477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:17.395494 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:17.398517 (+ 3023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:17.399357 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:17.399553 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:17.405454 (+ 5901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:17.405464 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:17.405471 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:17.405524 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:17.409813 (+ 4289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:17.411331 (+ 1518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:17.411396 (+ 65us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:17.411400 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:17.411599 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:17.412499 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:17.412508 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:17.412523 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:17.413147 (+ 624us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:17.413313 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:17.416428 (+ 3115us) server_negotiation.cc:299] Negotiation successful
0928 14:56:17.416749 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2253,"thread_start_us":2144,"threads_started":1}
I20260928 14:56:17.417496 952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:17.388171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34279 (local address 127.31.40.3:58015)
0928 14:56:17.389029 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:17.389076 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:17.391221 (+ 2145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:17.392194 (+ 973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:17.392207 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:17.393141 (+ 934us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:56:17.394165 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:17.394194 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:17.395677 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:17.395681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:17.398309 (+ 2628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:17.398330 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:17.399663 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:17.399675 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:17.400108 (+ 433us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:17.401062 (+ 954us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:56:17.401123 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:56:17.405097 (+ 3974us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:56:17.410093 (+ 4996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:17.410106 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:17.410136 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:17.411065 (+ 929us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:17.411804 (+ 739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:17.411812 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:17.411819 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:17.412266 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:17.413401 (+ 1135us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:56:17.413415 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:56:17.414101 (+ 686us) client_negotiation.cc:769] Sending connection context
0928 14:56:17.414509 (+ 408us) client_negotiation.cc:240] Negotiation successful
0928 14:56:17.415297 (+ 788us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":356,"threads_started":1}
I20260928 14:56:17.441439 927 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.3:34307
I20260928 14:56:17.441905 1063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.3:34307 every 8 connection(s)
I20260928 14:56:17.443395 927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
I20260928 14:56:17.444562 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 927
I20260928 14:56:17.444806 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/ts-2/wal/instance
I20260928 14:56:17.445387 31904 external_mini_cluster.cc:1428] Setting key e834e0ef60ffecf93b205b81cbcba8fa
I20260928 14:56:17.446375 1064 heartbeater.cc:347] Connected to a master server at 127.31.40.62:34279
I20260928 14:56:17.446811 1064 heartbeater.cc:464] Registering TS with master...
I20260928 14:56:17.447731 1064 heartbeater.cc:511] Master 127.31.40.62:34279 requested a full tablet report, sending...
I20260928 14:56:17.449311 574 ts_manager.cc:194] Registered new tserver with Master: 372c38dcee1c43e1af320d7a26511d8c (127.31.40.3:34307)
I20260928 14:56:17.450172 574 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.3@KRBTEST.COM'} at 127.31.40.3:58015
I20260928 14:56:17.450680 951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:17.431819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:51891 (local address 127.31.40.62:34279)
0928 14:56:17.432744 (+ 925us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:17.432754 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:17.432790 (+ 36us) server_negotiation.cc:407] Connection header received
0928 14:56:17.433041 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:17.433050 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:17.433220 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:17.435991 (+ 2771us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:17.436244 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:17.437405 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:17.439734 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:17.440617 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:17.440793 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:17.443489 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:17.443499 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:17.443506 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:17.443552 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:17.447435 (+ 3883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:17.448387 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:17.448395 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:17.448398 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:17.448555 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:17.449123 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:17.449131 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:17.449136 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:17.449504 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:17.449668 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:17.450049 (+ 381us) server_negotiation.cc:299] Negotiation successful
0928 14:56:17.450282 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":715}
I20260928 14:56:17.450738 1053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:17.431831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34279 (local address 127.31.40.3:51891)
0928 14:56:17.432622 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:17.432646 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:17.432838 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:17.433516 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:17.433522 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:17.434522 (+ 1000us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:56:17.435088 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:17.435119 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:17.437611 (+ 2492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:17.437619 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:17.439513 (+ 1894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:17.439537 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:17.440886 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:17.440896 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:17.441165 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:17.441973 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:56:17.441996 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:56:17.443187 (+ 1191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:56:17.447755 (+ 4568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:17.447759 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:17.447762 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:17.448267 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:17.448846 (+ 579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:17.448850 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:17.448854 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:17.449012 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:17.449642 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:56:17.449647 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:56:17.449822 (+ 175us) client_negotiation.cc:769] Sending connection context
0928 14:56:17.449964 (+ 142us) client_negotiation.cc:240] Negotiation successful
0928 14:56:17.450296 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":445,"threads_started":1}
I20260928 14:56:17.461695 31904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 14:56:17 dist-test-slave-j968 krb5kdc[528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607377, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 14:56:17 dist-test-slave-j968 krb5kdc[528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607377, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:56:17.513060 951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:17.496176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57182 (local address 127.31.40.62:34279)
0928 14:56:17.496420 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:17.496425 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:17.496556 (+ 131us) server_negotiation.cc:407] Connection header received
0928 14:56:17.496855 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:17.496860 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:17.497041 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:17.497141 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:17.498620 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:17.499423 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:17.501476 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:17.502162 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:17.502283 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:17.506867 (+ 4584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:17.506873 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:17.506937 (+ 64us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:17.506983 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:17.509819 (+ 2836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:17.510655 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:17.510660 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:17.510662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:17.510826 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:17.511373 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:17.511377 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:17.511380 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:17.511693 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:17.512046 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:17.512513 (+ 467us) server_negotiation.cc:299] Negotiation successful
0928 14:56:17.512697 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 14:56:17.517787 574 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57182:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:56:17.525513 574 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:56:17.550905 1079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:17.540282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48030 (local address 127.31.40.2:36135)
0928 14:56:17.541809 (+ 1527us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:17.541823 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:17.541862 (+ 39us) server_negotiation.cc:407] Connection header received
0928 14:56:17.542022 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:17.542034 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:17.542320 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:17.542552 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:56:17.543295 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:17.545099 (+ 1804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:17.548096 (+ 2997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:17.549581 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:17.549754 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:17.550010 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:17.550344 (+ 334us) server_negotiation.cc:299] Negotiation successful
0928 14:56:17.550554 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1305,"thread_start_us":1361,"threads_started":1}
I20260928 14:56:17.550966 1078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:17.539889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.2:36135 (local address 127.0.0.1:48030)
0928 14:56:17.541368 (+ 1479us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:17.541385 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:17.541486 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:17.542563 (+ 1077us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:17.542571 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:17.542600 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:56:17.543095 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:17.543109 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:17.545291 (+ 2182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:17.545300 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:17.547826 (+ 2526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:17.547846 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:17.549815 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:17.549822 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:17.550053 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:17.550115 (+ 62us) client_negotiation.cc:769] Sending connection context
0928 14:56:17.550268 (+ 153us) client_negotiation.cc:240] Negotiation successful
0928 14:56:17.550515 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1353,"mutex_wait_us":1,"thread_start_us":512,"threads_started":1}
I20260928 14:56:17.552676 1081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:17.542594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:44729 (local address 127.0.0.1:59954)
0928 14:56:17.543248 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:17.543271 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:17.543423 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:17.544544 (+ 1121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:17.544549 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:17.544571 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:56:17.545106 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:17.545120 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:17.546799 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:17.546806 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:17.549503 (+ 2697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:17.549521 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:17.551639 (+ 2118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:17.551648 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:17.551908 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:17.551946 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 14:56:17.552063 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 14:56:17.552224 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"mutex_wait_us":67,"thread_start_us":277,"threads_started":1}
I20260928 14:56:17.553301 1082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:17.542906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59954 (local address 127.31.40.1:44729)
0928 14:56:17.543966 (+ 1060us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:17.543973 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:17.543994 (+ 21us) server_negotiation.cc:407] Connection header received
0928 14:56:17.544089 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:17.544096 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:17.544323 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:17.545379 (+ 1056us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:56:17.545583 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:17.546615 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:17.549708 (+ 3093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:17.551409 (+ 1701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:17.551566 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:17.551824 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:17.552138 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 14:56:17.552353 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":878,"thread_start_us":751,"threads_started":1}
I20260928 14:56:17.554241 1077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:17.539773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.3:34307 (local address 127.0.0.1:39810)
0928 14:56:17.540622 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:17.540669 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:17.540830 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:17.543811 (+ 2981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:17.543816 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:17.543841 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:56:17.544253 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:17.544266 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:17.547054 (+ 2788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:17.547063 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:17.549642 (+ 2579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:17.549653 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:17.552044 (+ 2391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:17.552050 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:17.553395 (+ 1345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:17.553437 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 14:56:17.553511 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 14:56:17.553779 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"mutex_wait_us":6,"thread_start_us":1329,"threads_started":1}
I20260928 14:56:17.554527 1080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:17.540149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39810 (local address 127.31.40.3:34307)
0928 14:56:17.543197 (+ 3048us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:17.543206 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:17.543239 (+ 33us) server_negotiation.cc:407] Connection header received
0928 14:56:17.543329 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:17.543336 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:17.543589 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:17.544448 (+ 859us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:56:17.544769 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:17.546875 (+ 2106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:17.549850 (+ 2975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:17.551562 (+ 1712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:17.551709 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:17.552011 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:17.554019 (+ 2008us) server_negotiation.cc:299] Negotiation successful
0928 14:56:17.554211 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2868,"thread_start_us":2016,"threads_started":1}
I20260928 14:56:17.556051 853 tablet_service.cc:1467] Processing CreateTablet for tablet 6de95c9761fe49d2b9619b1ef44b6f14 (DEFAULT_TABLE table=test-table [id=5d612671fce54ea8855730bcaa3abc5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:56:17.557102 712 tablet_service.cc:1467] Processing CreateTablet for tablet 6de95c9761fe49d2b9619b1ef44b6f14 (DEFAULT_TABLE table=test-table [id=5d612671fce54ea8855730bcaa3abc5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:56:17.557648 853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6de95c9761fe49d2b9619b1ef44b6f14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:56:17.558720 712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6de95c9761fe49d2b9619b1ef44b6f14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:56:17.558270 993 tablet_service.cc:1467] Processing CreateTablet for tablet 6de95c9761fe49d2b9619b1ef44b6f14 (DEFAULT_TABLE table=test-table [id=5d612671fce54ea8855730bcaa3abc5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:56:17.559597 993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6de95c9761fe49d2b9619b1ef44b6f14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:56:17.572919 1083 tablet_bootstrap.cc:492] T 6de95c9761fe49d2b9619b1ef44b6f14 P ac470d791ea04f708719e48abd721021: Bootstrap starting.
I20260928 14:56:17.575753 1084 tablet_bootstrap.cc:492] T 6de95c9761fe49d2b9619b1ef44b6f14 P a8d762c4f42841c186f48a56ecc5354f: Bootstrap starting.
I20260928 14:56:17.577485 1083 tablet_bootstrap.cc:654] T 6de95c9761fe49d2b9619b1ef44b6f14 P ac470d791ea04f708719e48abd721021: Neither blocks nor log segments found. Creating new log.
I20260928 14:56:17.577754 1085 tablet_bootstrap.cc:492] T 6de95c9761fe49d2b9619b1ef44b6f14 P 372c38dcee1c43e1af320d7a26511d8c: Bootstrap starting.
I20260928 14:56:17.579406 1083 log.cc:824] T 6de95c9761fe49d2b9619b1ef44b6f14 P ac470d791ea04f708719e48abd721021: Log is configured to *not* fsync() on all Append() calls
I20260928 14:56:17.580344 1084 tablet_bootstrap.cc:654] T 6de95c9761fe49d2b9619b1ef44b6f14 P a8d762c4f42841c186f48a56ecc5354f: Neither blocks nor log segments found. Creating new log.
I20260928 14:56:17.581326 1085 tablet_bootstrap.cc:654] T 6de95c9761fe49d2b9619b1ef44b6f14 P 372c38dcee1c43e1af320d7a26511d8c: Neither blocks nor log segments found. Creating new log.
I20260928 14:56:17.581815 1084 log.cc:824] T 6de95c9761fe49d2b9619b1ef44b6f14 P a8d762c4f42841c186f48a56ecc5354f: Log is configured to *not* fsync() on all Append() calls
I20260928 14:56:17.582664 1085 log.cc:824] T 6de95c9761fe49d2b9619b1ef44b6f14 P 372c38dcee1c43e1af320d7a26511d8c: Log is configured to *not* fsync() on all Append() calls
I20260928 14:56:17.583304 1083 tablet_bootstrap.cc:492] T 6de95c9761fe49d2b9619b1ef44b6f14 P ac470d791ea04f708719e48abd721021: No bootstrap required, opened a new log
I20260928 14:56:17.583716 1083 ts_tablet_manager.cc:1397] T 6de95c9761fe49d2b9619b1ef44b6f14 P ac470d791ea04f708719e48abd721021: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260928 14:56:17.585481 1084 tablet_bootstrap.cc:492] T 6de95c9761fe49d2b9619b1ef44b6f14 P a8d762c4f42841c186f48a56ecc5354f: No bootstrap required, opened a new log
I20260928 14:56:17.585794 1084 ts_tablet_manager.cc:1397] T 6de95c9761fe49d2b9619b1ef44b6f14 P a8d762c4f42841c186f48a56ecc5354f: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260928 14:56:17.587991 1085 tablet_bootstrap.cc:492] T 6de95c9761fe49d2b9619b1ef44b6f14 P 372c38dcee1c43e1af320d7a26511d8c: No bootstrap required, opened a new log
I20260928 14:56:17.588320 1085 ts_tablet_manager.cc:1397] T 6de95c9761fe49d2b9619b1ef44b6f14 P 372c38dcee1c43e1af320d7a26511d8c: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.003s
I20260928 14:56:17.596093 1085 raft_consensus.cc:348] T 6de95c9761fe49d2b9619b1ef44b6f14 P 372c38dcee1c43e1af320d7a26511d8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372c38dcee1c43e1af320d7a26511d8c" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34307 } } peers { permanent_uuid: "a8d762c4f42841c186f48a56ecc5354f" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 36135 } } peers { permanent_uuid: "ac470d791ea04f708719e48abd721021" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 44729 } }
I20260928 14:56:17.596545 1085 raft_consensus.cc:374] T 6de95c9761fe49d2b9619b1ef44b6f14 P 372c38dcee1c43e1af320d7a26511d8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:56:17.596673 1085 raft_consensus.cc:729] T 6de95c9761fe49d2b9619b1ef44b6f14 P 372c38dcee1c43e1af320d7a26511d8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 372c38dcee1c43e1af320d7a26511d8c, State: Initialized, Role: FOLLOWER
I20260928 14:56:17.596606 1083 raft_consensus.cc:348] T 6de95c9761fe49d2b9619b1ef44b6f14 P ac470d791ea04f708719e48abd721021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372c38dcee1c43e1af320d7a26511d8c" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34307 } } peers { permanent_uuid: "a8d762c4f42841c186f48a56ecc5354f" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 36135 } } peers { permanent_uuid: "ac470d791ea04f708719e48abd721021" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 44729 } }
I20260928 14:56:17.597246 1083 raft_consensus.cc:374] T 6de95c9761fe49d2b9619b1ef44b6f14 P ac470d791ea04f708719e48abd721021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:56:17.597373 1083 raft_consensus.cc:729] T 6de95c9761fe49d2b9619b1ef44b6f14 P ac470d791ea04f708719e48abd721021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac470d791ea04f708719e48abd721021, State: Initialized, Role: FOLLOWER
I20260928 14:56:17.597378 1085 consensus_queue.cc:260] T 6de95c9761fe49d2b9619b1ef44b6f14 P 372c38dcee1c43e1af320d7a26511d8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372c38dcee1c43e1af320d7a26511d8c" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34307 } } peers { permanent_uuid: "a8d762c4f42841c186f48a56ecc5354f" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 36135 } } peers { permanent_uuid: "ac470d791ea04f708719e48abd721021" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 44729 } }
I20260928 14:56:17.598206 1083 consensus_queue.cc:260] T 6de95c9761fe49d2b9619b1ef44b6f14 P ac470d791ea04f708719e48abd721021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372c38dcee1c43e1af320d7a26511d8c" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34307 } } peers { permanent_uuid: "a8d762c4f42841c186f48a56ecc5354f" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 36135 } } peers { permanent_uuid: "ac470d791ea04f708719e48abd721021" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 44729 } }
I20260928 14:56:17.598747 1084 raft_consensus.cc:348] T 6de95c9761fe49d2b9619b1ef44b6f14 P a8d762c4f42841c186f48a56ecc5354f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372c38dcee1c43e1af320d7a26511d8c" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34307 } } peers { permanent_uuid: "a8d762c4f42841c186f48a56ecc5354f" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 36135 } } peers { permanent_uuid: "ac470d791ea04f708719e48abd721021" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 44729 } }
I20260928 14:56:17.599412 1085 ts_tablet_manager.cc:1428] T 6de95c9761fe49d2b9619b1ef44b6f14 P 372c38dcee1c43e1af320d7a26511d8c: Time spent starting tablet: real 0.011s user 0.009s sys 0.001s
I20260928 14:56:17.599449 1084 raft_consensus.cc:374] T 6de95c9761fe49d2b9619b1ef44b6f14 P a8d762c4f42841c186f48a56ecc5354f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:56:17.599666 1084 raft_consensus.cc:729] T 6de95c9761fe49d2b9619b1ef44b6f14 P a8d762c4f42841c186f48a56ecc5354f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8d762c4f42841c186f48a56ecc5354f, State: Initialized, Role: FOLLOWER
I20260928 14:56:17.600006 1064 heartbeater.cc:503] Master 127.31.40.62:34279 was elected leader, sending a full tablet report...
I20260928 14:56:17.600807 1084 consensus_queue.cc:260] T 6de95c9761fe49d2b9619b1ef44b6f14 P a8d762c4f42841c186f48a56ecc5354f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372c38dcee1c43e1af320d7a26511d8c" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34307 } } peers { permanent_uuid: "a8d762c4f42841c186f48a56ecc5354f" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 36135 } } peers { permanent_uuid: "ac470d791ea04f708719e48abd721021" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 44729 } }
I20260928 14:56:17.602562 923 heartbeater.cc:503] Master 127.31.40.62:34279 was elected leader, sending a full tablet report...
I20260928 14:56:17.603396 1084 ts_tablet_manager.cc:1428] T 6de95c9761fe49d2b9619b1ef44b6f14 P a8d762c4f42841c186f48a56ecc5354f: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
I20260928 14:56:17.605221 1083 ts_tablet_manager.cc:1428] T 6de95c9761fe49d2b9619b1ef44b6f14 P ac470d791ea04f708719e48abd721021: Time spent starting tablet: real 0.021s user 0.011s sys 0.008s
W20260928 14:56:17.650971 924 tablet.cc:2369] T 6de95c9761fe49d2b9619b1ef44b6f14 P a8d762c4f42841c186f48a56ecc5354f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:56:17.695489 1065 tablet.cc:2369] T 6de95c9761fe49d2b9619b1ef44b6f14 P 372c38dcee1c43e1af320d7a26511d8c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:56:17.749956 1089 raft_consensus.cc:482] T 6de95c9761fe49d2b9619b1ef44b6f14 P 372c38dcee1c43e1af320d7a26511d8c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:56:17.750321 1089 raft_consensus.cc:504] T 6de95c9761fe49d2b9619b1ef44b6f14 P 372c38dcee1c43e1af320d7a26511d8c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372c38dcee1c43e1af320d7a26511d8c" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34307 } } peers { permanent_uuid: "a8d762c4f42841c186f48a56ecc5354f" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 36135 } } peers { permanent_uuid: "ac470d791ea04f708719e48abd721021" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 44729 } }
I20260928 14:56:17.752117 1089 leader_election.cc:288] T 6de95c9761fe49d2b9619b1ef44b6f14 P 372c38dcee1c43e1af320d7a26511d8c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8d762c4f42841c186f48a56ecc5354f (127.31.40.2:36135), ac470d791ea04f708719e48abd721021 (127.31.40.1:44729)
I20260928 14:56:17.759517 952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:17.752495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.2:36135 (local address 127.31.40.3:60445)
0928 14:56:17.752729 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:17.752750 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:17.752928 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:17.753396 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:17.753403 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:17.753440 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:56:17.753705 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:17.753725 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:17.755054 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:17.755058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:17.757283 (+ 2225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:17.757302 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:17.758881 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:17.758888 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:17.759048 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:17.759075 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 14:56:17.759169 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 14:56:17.759274 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":26}
I20260928 14:56:17.759673 1079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:17.752566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:60445 (local address 127.31.40.2:36135)
0928 14:56:17.752812 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:17.752820 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:17.752843 (+ 23us) server_negotiation.cc:407] Connection header received
0928 14:56:17.753113 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:17.753121 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:17.753225 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:17.753333 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:56:17.753900 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:17.754839 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:17.757530 (+ 2691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:17.758679 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:17.758834 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:17.758949 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:17.759237 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 14:56:17.759390 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 14:56:17.760754 1092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:17.752910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:44729 (local address 127.31.40.3:41763)
0928 14:56:17.753740 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:17.753760 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:17.753886 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:17.754290 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:17.754293 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:17.754313 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:56:17.754666 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:17.754674 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:17.755713 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:17.755719 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:17.758013 (+ 2294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:17.758025 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:17.760013 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:17.760021 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:17.760205 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:17.760249 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 14:56:17.760356 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 14:56:17.760463 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"mutex_wait_us":2,"thread_start_us":525,"threads_started":1}
I20260928 14:56:17.760870 1082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:17.753222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:41763 (local address 127.31.40.1:44729)
0928 14:56:17.753421 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:17.753427 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:17.753854 (+ 427us) server_negotiation.cc:407] Connection header received
0928 14:56:17.754025 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:17.754031 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:17.754168 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:17.754269 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:56:17.754826 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:17.755561 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:17.758208 (+ 2647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:17.759516 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:17.759660 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:17.759782 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:17.760460 (+ 678us) server_negotiation.cc:299] Negotiation successful
0928 14:56:17.760625 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 14:56:17.761053 878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6de95c9761fe49d2b9619b1ef44b6f14" candidate_uuid: "372c38dcee1c43e1af320d7a26511d8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8d762c4f42841c186f48a56ecc5354f" is_pre_election: true
I20260928 14:56:17.761667 878 raft_consensus.cc:2454] T 6de95c9761fe49d2b9619b1ef44b6f14 P a8d762c4f42841c186f48a56ecc5354f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 372c38dcee1c43e1af320d7a26511d8c in term 0.
I20260928 14:56:17.761893 732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6de95c9761fe49d2b9619b1ef44b6f14" candidate_uuid: "372c38dcee1c43e1af320d7a26511d8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac470d791ea04f708719e48abd721021" is_pre_election: true
I20260928 14:56:17.762457 946 leader_election.cc:302] T 6de95c9761fe49d2b9619b1ef44b6f14 P 372c38dcee1c43e1af320d7a26511d8c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 372c38dcee1c43e1af320d7a26511d8c, a8d762c4f42841c186f48a56ecc5354f; no voters:
I20260928 14:56:17.762588 732 raft_consensus.cc:2454] T 6de95c9761fe49d2b9619b1ef44b6f14 P ac470d791ea04f708719e48abd721021 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 372c38dcee1c43e1af320d7a26511d8c in term 0.
I20260928 14:56:17.762925 1089 raft_consensus.cc:2781] T 6de95c9761fe49d2b9619b1ef44b6f14 P 372c38dcee1c43e1af320d7a26511d8c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:56:17.763137 1089 raft_consensus.cc:482] T 6de95c9761fe49d2b9619b1ef44b6f14 P 372c38dcee1c43e1af320d7a26511d8c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:56:17.763284 1089 raft_consensus.cc:3037] T 6de95c9761fe49d2b9619b1ef44b6f14 P 372c38dcee1c43e1af320d7a26511d8c [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:56:17.765730 1089 raft_consensus.cc:504] T 6de95c9761fe49d2b9619b1ef44b6f14 P 372c38dcee1c43e1af320d7a26511d8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372c38dcee1c43e1af320d7a26511d8c" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34307 } } peers { permanent_uuid: "a8d762c4f42841c186f48a56ecc5354f" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 36135 } } peers { permanent_uuid: "ac470d791ea04f708719e48abd721021" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 44729 } }
I20260928 14:56:17.766495 1089 leader_election.cc:288] T 6de95c9761fe49d2b9619b1ef44b6f14 P 372c38dcee1c43e1af320d7a26511d8c [CANDIDATE]: Term 1 election: Requested vote from peers a8d762c4f42841c186f48a56ecc5354f (127.31.40.2:36135), ac470d791ea04f708719e48abd721021 (127.31.40.1:44729)
I20260928 14:56:17.767073 878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6de95c9761fe49d2b9619b1ef44b6f14" candidate_uuid: "372c38dcee1c43e1af320d7a26511d8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8d762c4f42841c186f48a56ecc5354f"
I20260928 14:56:17.767088 732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6de95c9761fe49d2b9619b1ef44b6f14" candidate_uuid: "372c38dcee1c43e1af320d7a26511d8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac470d791ea04f708719e48abd721021"
I20260928 14:56:17.767342 732 raft_consensus.cc:3037] T 6de95c9761fe49d2b9619b1ef44b6f14 P ac470d791ea04f708719e48abd721021 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:56:17.767340 878 raft_consensus.cc:3037] T 6de95c9761fe49d2b9619b1ef44b6f14 P a8d762c4f42841c186f48a56ecc5354f [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:56:17.770803 732 raft_consensus.cc:2454] T 6de95c9761fe49d2b9619b1ef44b6f14 P ac470d791ea04f708719e48abd721021 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 372c38dcee1c43e1af320d7a26511d8c in term 1.
I20260928 14:56:17.770805 878 raft_consensus.cc:2454] T 6de95c9761fe49d2b9619b1ef44b6f14 P a8d762c4f42841c186f48a56ecc5354f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 372c38dcee1c43e1af320d7a26511d8c in term 1.
I20260928 14:56:17.771329 946 leader_election.cc:302] T 6de95c9761fe49d2b9619b1ef44b6f14 P 372c38dcee1c43e1af320d7a26511d8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 372c38dcee1c43e1af320d7a26511d8c, ac470d791ea04f708719e48abd721021; no voters:
I20260928 14:56:17.771663 1089 raft_consensus.cc:2781] T 6de95c9761fe49d2b9619b1ef44b6f14 P 372c38dcee1c43e1af320d7a26511d8c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:56:17.772231 1089 raft_consensus.cc:686] T 6de95c9761fe49d2b9619b1ef44b6f14 P 372c38dcee1c43e1af320d7a26511d8c [term 1 LEADER]: Becoming Leader. State: Replica: 372c38dcee1c43e1af320d7a26511d8c, State: Running, Role: LEADER
I20260928 14:56:17.772784 1089 consensus_queue.cc:237] T 6de95c9761fe49d2b9619b1ef44b6f14 P 372c38dcee1c43e1af320d7a26511d8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372c38dcee1c43e1af320d7a26511d8c" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34307 } } peers { permanent_uuid: "a8d762c4f42841c186f48a56ecc5354f" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 36135 } } peers { permanent_uuid: "ac470d791ea04f708719e48abd721021" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 44729 } }
I20260928 14:56:17.778513 573 catalog_manager.cc:5495] T 6de95c9761fe49d2b9619b1ef44b6f14 P 372c38dcee1c43e1af320d7a26511d8c reported cstate change: term changed from 0 to 1, leader changed from <none> to 372c38dcee1c43e1af320d7a26511d8c (127.31.40.3). New cstate: current_term: 1 leader_uuid: "372c38dcee1c43e1af320d7a26511d8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372c38dcee1c43e1af320d7a26511d8c" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8d762c4f42841c186f48a56ecc5354f" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 36135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac470d791ea04f708719e48abd721021" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 44729 } health_report { overall_health: UNKNOWN } } }
I20260928 14:56:17.806838 1080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:17.800874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39820 (local address 127.31.40.3:34307)
0928 14:56:17.801104 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:17.801111 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:17.801365 (+ 254us) server_negotiation.cc:407] Connection header received
0928 14:56:17.801725 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:17.801729 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:17.801807 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:17.801940 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:56:17.802343 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:17.803160 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:17.804563 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:17.805013 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:17.805132 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:17.805572 (+ 440us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:56:17.806141 (+ 569us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:56:17.806257 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:17.806441 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 14:56:17.806556 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 14:56:17.816069 732 raft_consensus.cc:1261] T 6de95c9761fe49d2b9619b1ef44b6f14 P ac470d791ea04f708719e48abd721021 [term 1 FOLLOWER]: Refusing update from remote peer 372c38dcee1c43e1af320d7a26511d8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:56:17.816069 878 raft_consensus.cc:1261] T 6de95c9761fe49d2b9619b1ef44b6f14 P a8d762c4f42841c186f48a56ecc5354f [term 1 FOLLOWER]: Refusing update from remote peer 372c38dcee1c43e1af320d7a26511d8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:56:17.817077 1093 consensus_queue.cc:1035] T 6de95c9761fe49d2b9619b1ef44b6f14 P 372c38dcee1c43e1af320d7a26511d8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac470d791ea04f708719e48abd721021" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 44729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:56:17.817449 1089 consensus_queue.cc:1035] T 6de95c9761fe49d2b9619b1ef44b6f14 P 372c38dcee1c43e1af320d7a26511d8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8d762c4f42841c186f48a56ecc5354f" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 36135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 14:56:17.839267 783 tablet.cc:2369] T 6de95c9761fe49d2b9619b1ef44b6f14 P ac470d791ea04f708719e48abd721021: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:56:17.838008 1097 mvcc.cc:205] Tried to move back new op lower bound from 7334327819523551232 to 7334327819362246656. Current Snapshot: MvccSnapshot[applied={T|T < 7334327819523551232}]
I20260928 14:56:17.846868 1098 mvcc.cc:205] Tried to move back new op lower bound from 7334327819523551232 to 7334327819362246656. Current Snapshot: MvccSnapshot[applied={T|T < 7334327819523551232}]
I20260928 14:56:17.860314 573 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:56:17.860630 573 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:56:17.864390 573 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 633f7fb706e243119cb41b0ad22c5be6: Sending DeleteTablet for 3 replicas of tablet 6de95c9761fe49d2b9619b1ef44b6f14
I20260928 14:56:17.865723 993 tablet_service.cc:1514] Processing DeleteTablet for tablet 6de95c9761fe49d2b9619b1ef44b6f14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:56:17 UTC) from {username='kudu', principal='kudu/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:39810
I20260928 14:56:17.866133 853 tablet_service.cc:1514] Processing DeleteTablet for tablet 6de95c9761fe49d2b9619b1ef44b6f14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:56:17 UTC) from {username='kudu', principal='kudu/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:48030
I20260928 14:56:17.866887 712 tablet_service.cc:1514] Processing DeleteTablet for tablet 6de95c9761fe49d2b9619b1ef44b6f14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:56:17 UTC) from {username='kudu', principal='kudu/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:59954
I20260928 14:56:17.869863 573 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57182:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:56:17.871794 573 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:56:17.881155 1107 tablet_replica.cc:330] T 6de95c9761fe49d2b9619b1ef44b6f14 P a8d762c4f42841c186f48a56ecc5354f: stopping tablet replica
I20260928 14:56:17.881346 992 tablet_service.cc:1467] Processing CreateTablet for tablet 603775196acc416da4295236a90ab22f (DEFAULT_TABLE table=test-table [id=1098dd7f638d472ab27285a340a53950]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:56:17.881736 1107 raft_consensus.cc:2229] T 6de95c9761fe49d2b9619b1ef44b6f14 P a8d762c4f42841c186f48a56ecc5354f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:56:17.881950 992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 603775196acc416da4295236a90ab22f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:56:17.882126 852 tablet_service.cc:1467] Processing CreateTablet for tablet 603775196acc416da4295236a90ab22f (DEFAULT_TABLE table=test-table [id=1098dd7f638d472ab27285a340a53950]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:56:17.882329 1107 raft_consensus.cc:2258] T 6de95c9761fe49d2b9619b1ef44b6f14 P a8d762c4f42841c186f48a56ecc5354f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:56:17.882792 852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 603775196acc416da4295236a90ab22f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:56:17.885525 1108 tablet_replica.cc:330] T 6de95c9761fe49d2b9619b1ef44b6f14 P ac470d791ea04f708719e48abd721021: stopping tablet replica
I20260928 14:56:17.886099 1108 raft_consensus.cc:2229] T 6de95c9761fe49d2b9619b1ef44b6f14 P ac470d791ea04f708719e48abd721021 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:56:17.886641 1108 raft_consensus.cc:2258] T 6de95c9761fe49d2b9619b1ef44b6f14 P ac470d791ea04f708719e48abd721021 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:56:17.887498 1106 tablet_replica.cc:330] T 6de95c9761fe49d2b9619b1ef44b6f14 P 372c38dcee1c43e1af320d7a26511d8c: stopping tablet replica
I20260928 14:56:17.887923 711 tablet_service.cc:1467] Processing CreateTablet for tablet 603775196acc416da4295236a90ab22f (DEFAULT_TABLE table=test-table [id=1098dd7f638d472ab27285a340a53950]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:56:17.888000 1106 raft_consensus.cc:2229] T 6de95c9761fe49d2b9619b1ef44b6f14 P 372c38dcee1c43e1af320d7a26511d8c [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:56:17.888410 711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 603775196acc416da4295236a90ab22f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:56:17.888756 1106 raft_consensus.cc:2258] T 6de95c9761fe49d2b9619b1ef44b6f14 P 372c38dcee1c43e1af320d7a26511d8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:56:17.889104 1108 ts_tablet_manager.cc:1905] T 6de95c9761fe49d2b9619b1ef44b6f14 P ac470d791ea04f708719e48abd721021: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:56:17.891376 1085 tablet_bootstrap.cc:492] T 603775196acc416da4295236a90ab22f P 372c38dcee1c43e1af320d7a26511d8c: Bootstrap starting.
I20260928 14:56:17.892127 1084 tablet_bootstrap.cc:492] T 603775196acc416da4295236a90ab22f P a8d762c4f42841c186f48a56ecc5354f: Bootstrap starting.
I20260928 14:56:17.893124 1085 tablet_bootstrap.cc:654] T 603775196acc416da4295236a90ab22f P 372c38dcee1c43e1af320d7a26511d8c: Neither blocks nor log segments found. Creating new log.
I20260928 14:56:17.893770 1084 tablet_bootstrap.cc:654] T 603775196acc416da4295236a90ab22f P a8d762c4f42841c186f48a56ecc5354f: Neither blocks nor log segments found. Creating new log.
I20260928 14:56:17.895774 1106 ts_tablet_manager.cc:1905] T 6de95c9761fe49d2b9619b1ef44b6f14 P 372c38dcee1c43e1af320d7a26511d8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:56:17.899576 1107 ts_tablet_manager.cc:1905] T 6de95c9761fe49d2b9619b1ef44b6f14 P a8d762c4f42841c186f48a56ecc5354f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:56:17.903473 1106 ts_tablet_manager.cc:1918] T 6de95c9761fe49d2b9619b1ef44b6f14 P 372c38dcee1c43e1af320d7a26511d8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:56:17.903663 1106 log.cc:1196] T 6de95c9761fe49d2b9619b1ef44b6f14 P 372c38dcee1c43e1af320d7a26511d8c: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/ts-2/wal/wals/6de95c9761fe49d2b9619b1ef44b6f14
I20260928 14:56:17.904202 1083 tablet_bootstrap.cc:492] T 603775196acc416da4295236a90ab22f P ac470d791ea04f708719e48abd721021: Bootstrap starting.
I20260928 14:56:17.904218 1106 ts_tablet_manager.cc:1939] T 6de95c9761fe49d2b9619b1ef44b6f14 P 372c38dcee1c43e1af320d7a26511d8c: Deleting consensus metadata
I20260928 14:56:17.905983 1083 tablet_bootstrap.cc:654] T 603775196acc416da4295236a90ab22f P ac470d791ea04f708719e48abd721021: Neither blocks nor log segments found. Creating new log.
I20260928 14:56:17.906126 559 catalog_manager.cc:4841] TS 372c38dcee1c43e1af320d7a26511d8c (127.31.40.3:34307): tablet 6de95c9761fe49d2b9619b1ef44b6f14 (table test-table [id=5d612671fce54ea8855730bcaa3abc5f]) successfully deleted
I20260928 14:56:17.907856 1085 tablet_bootstrap.cc:492] T 603775196acc416da4295236a90ab22f P 372c38dcee1c43e1af320d7a26511d8c: No bootstrap required, opened a new log
I20260928 14:56:17.908293 1085 ts_tablet_manager.cc:1397] T 603775196acc416da4295236a90ab22f P 372c38dcee1c43e1af320d7a26511d8c: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20260928 14:56:17.909992 1085 raft_consensus.cc:348] T 603775196acc416da4295236a90ab22f P 372c38dcee1c43e1af320d7a26511d8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac470d791ea04f708719e48abd721021" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 44729 } } peers { permanent_uuid: "a8d762c4f42841c186f48a56ecc5354f" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 36135 } } peers { permanent_uuid: "372c38dcee1c43e1af320d7a26511d8c" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34307 } }
I20260928 14:56:17.910660 1085 raft_consensus.cc:374] T 603775196acc416da4295236a90ab22f P 372c38dcee1c43e1af320d7a26511d8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:56:17.910975 1085 raft_consensus.cc:729] T 603775196acc416da4295236a90ab22f P 372c38dcee1c43e1af320d7a26511d8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 372c38dcee1c43e1af320d7a26511d8c, State: Initialized, Role: FOLLOWER
I20260928 14:56:17.913296 1108 ts_tablet_manager.cc:1918] T 6de95c9761fe49d2b9619b1ef44b6f14 P ac470d791ea04f708719e48abd721021: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:56:17.913431 1083 tablet_bootstrap.cc:492] T 603775196acc416da4295236a90ab22f P ac470d791ea04f708719e48abd721021: No bootstrap required, opened a new log
I20260928 14:56:17.913518 1108 log.cc:1196] T 6de95c9761fe49d2b9619b1ef44b6f14 P ac470d791ea04f708719e48abd721021: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/ts-0/wal/wals/6de95c9761fe49d2b9619b1ef44b6f14
I20260928 14:56:17.913657 1083 ts_tablet_manager.cc:1397] T 603775196acc416da4295236a90ab22f P ac470d791ea04f708719e48abd721021: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260928 14:56:17.914445 1108 ts_tablet_manager.cc:1939] T 6de95c9761fe49d2b9619b1ef44b6f14 P ac470d791ea04f708719e48abd721021: Deleting consensus metadata
I20260928 14:56:17.914039 1085 consensus_queue.cc:260] T 603775196acc416da4295236a90ab22f P 372c38dcee1c43e1af320d7a26511d8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac470d791ea04f708719e48abd721021" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 44729 } } peers { permanent_uuid: "a8d762c4f42841c186f48a56ecc5354f" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 36135 } } peers { permanent_uuid: "372c38dcee1c43e1af320d7a26511d8c" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34307 } }
I20260928 14:56:17.915266 1083 raft_consensus.cc:348] T 603775196acc416da4295236a90ab22f P ac470d791ea04f708719e48abd721021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac470d791ea04f708719e48abd721021" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 44729 } } peers { permanent_uuid: "a8d762c4f42841c186f48a56ecc5354f" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 36135 } } peers { permanent_uuid: "372c38dcee1c43e1af320d7a26511d8c" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34307 } }
I20260928 14:56:17.915573 1083 raft_consensus.cc:374] T 603775196acc416da4295236a90ab22f P ac470d791ea04f708719e48abd721021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:56:17.915792 1083 raft_consensus.cc:729] T 603775196acc416da4295236a90ab22f P ac470d791ea04f708719e48abd721021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac470d791ea04f708719e48abd721021, State: Initialized, Role: FOLLOWER
I20260928 14:56:17.916364 1083 consensus_queue.cc:260] T 603775196acc416da4295236a90ab22f P ac470d791ea04f708719e48abd721021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac470d791ea04f708719e48abd721021" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 44729 } } peers { permanent_uuid: "a8d762c4f42841c186f48a56ecc5354f" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 36135 } } peers { permanent_uuid: "372c38dcee1c43e1af320d7a26511d8c" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34307 } }
I20260928 14:56:17.916988 561 catalog_manager.cc:4841] TS ac470d791ea04f708719e48abd721021 (127.31.40.1:44729): tablet 6de95c9761fe49d2b9619b1ef44b6f14 (table test-table [id=5d612671fce54ea8855730bcaa3abc5f]) successfully deleted
I20260928 14:56:17.917929 1083 ts_tablet_manager.cc:1428] T 603775196acc416da4295236a90ab22f P ac470d791ea04f708719e48abd721021: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260928 14:56:17.919039 1085 ts_tablet_manager.cc:1428] T 603775196acc416da4295236a90ab22f P 372c38dcee1c43e1af320d7a26511d8c: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20260928 14:56:17.924270 1084 tablet_bootstrap.cc:492] T 603775196acc416da4295236a90ab22f P a8d762c4f42841c186f48a56ecc5354f: No bootstrap required, opened a new log
I20260928 14:56:17.924547 1084 ts_tablet_manager.cc:1397] T 603775196acc416da4295236a90ab22f P a8d762c4f42841c186f48a56ecc5354f: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.000s
I20260928 14:56:17.925848 1107 ts_tablet_manager.cc:1918] T 6de95c9761fe49d2b9619b1ef44b6f14 P a8d762c4f42841c186f48a56ecc5354f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:56:17.926158 1107 log.cc:1196] T 6de95c9761fe49d2b9619b1ef44b6f14 P a8d762c4f42841c186f48a56ecc5354f: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/ts-1/wal/wals/6de95c9761fe49d2b9619b1ef44b6f14
I20260928 14:56:17.926129 1084 raft_consensus.cc:348] T 603775196acc416da4295236a90ab22f P a8d762c4f42841c186f48a56ecc5354f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac470d791ea04f708719e48abd721021" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 44729 } } peers { permanent_uuid: "a8d762c4f42841c186f48a56ecc5354f" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 36135 } } peers { permanent_uuid: "372c38dcee1c43e1af320d7a26511d8c" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34307 } }
I20260928 14:56:17.926450 1084 raft_consensus.cc:374] T 603775196acc416da4295236a90ab22f P a8d762c4f42841c186f48a56ecc5354f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:56:17.926596 1084 raft_consensus.cc:729] T 603775196acc416da4295236a90ab22f P a8d762c4f42841c186f48a56ecc5354f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8d762c4f42841c186f48a56ecc5354f, State: Initialized, Role: FOLLOWER
I20260928 14:56:17.926924 1084 consensus_queue.cc:260] T 603775196acc416da4295236a90ab22f P a8d762c4f42841c186f48a56ecc5354f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac470d791ea04f708719e48abd721021" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 44729 } } peers { permanent_uuid: "a8d762c4f42841c186f48a56ecc5354f" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 36135 } } peers { permanent_uuid: "372c38dcee1c43e1af320d7a26511d8c" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34307 } }
I20260928 14:56:17.927351 1107 ts_tablet_manager.cc:1939] T 6de95c9761fe49d2b9619b1ef44b6f14 P a8d762c4f42841c186f48a56ecc5354f: Deleting consensus metadata
I20260928 14:56:17.928323 1084 ts_tablet_manager.cc:1428] T 603775196acc416da4295236a90ab22f P a8d762c4f42841c186f48a56ecc5354f: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260928 14:56:17.930122 561 catalog_manager.cc:4841] TS a8d762c4f42841c186f48a56ecc5354f (127.31.40.2:36135): tablet 6de95c9761fe49d2b9619b1ef44b6f14 (table test-table [id=5d612671fce54ea8855730bcaa3abc5f]) successfully deleted
I20260928 14:56:17.945096 1103 raft_consensus.cc:482] T 603775196acc416da4295236a90ab22f P 372c38dcee1c43e1af320d7a26511d8c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:56:17.945516 1103 raft_consensus.cc:504] T 603775196acc416da4295236a90ab22f P 372c38dcee1c43e1af320d7a26511d8c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac470d791ea04f708719e48abd721021" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 44729 } } peers { permanent_uuid: "a8d762c4f42841c186f48a56ecc5354f" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 36135 } } peers { permanent_uuid: "372c38dcee1c43e1af320d7a26511d8c" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34307 } }
I20260928 14:56:17.946311 1103 leader_election.cc:288] T 603775196acc416da4295236a90ab22f P 372c38dcee1c43e1af320d7a26511d8c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac470d791ea04f708719e48abd721021 (127.31.40.1:44729), a8d762c4f42841c186f48a56ecc5354f (127.31.40.2:36135)
I20260928 14:56:17.946998 878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "603775196acc416da4295236a90ab22f" candidate_uuid: "372c38dcee1c43e1af320d7a26511d8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8d762c4f42841c186f48a56ecc5354f" is_pre_election: true
I20260928 14:56:17.947126 732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "603775196acc416da4295236a90ab22f" candidate_uuid: "372c38dcee1c43e1af320d7a26511d8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac470d791ea04f708719e48abd721021" is_pre_election: true
I20260928 14:56:17.947266 878 raft_consensus.cc:2454] T 603775196acc416da4295236a90ab22f P a8d762c4f42841c186f48a56ecc5354f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 372c38dcee1c43e1af320d7a26511d8c in term 0.
I20260928 14:56:17.947430 732 raft_consensus.cc:2454] T 603775196acc416da4295236a90ab22f P ac470d791ea04f708719e48abd721021 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 372c38dcee1c43e1af320d7a26511d8c in term 0.
I20260928 14:56:17.947773 946 leader_election.cc:302] T 603775196acc416da4295236a90ab22f P 372c38dcee1c43e1af320d7a26511d8c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 372c38dcee1c43e1af320d7a26511d8c, a8d762c4f42841c186f48a56ecc5354f; no voters:
I20260928 14:56:17.948143 1103 raft_consensus.cc:2781] T 603775196acc416da4295236a90ab22f P 372c38dcee1c43e1af320d7a26511d8c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:56:17.948279 1103 raft_consensus.cc:482] T 603775196acc416da4295236a90ab22f P 372c38dcee1c43e1af320d7a26511d8c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:56:17.948397 1103 raft_consensus.cc:3037] T 603775196acc416da4295236a90ab22f P 372c38dcee1c43e1af320d7a26511d8c [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:56:17.950676 1103 raft_consensus.cc:504] T 603775196acc416da4295236a90ab22f P 372c38dcee1c43e1af320d7a26511d8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac470d791ea04f708719e48abd721021" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 44729 } } peers { permanent_uuid: "a8d762c4f42841c186f48a56ecc5354f" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 36135 } } peers { permanent_uuid: "372c38dcee1c43e1af320d7a26511d8c" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34307 } }
I20260928 14:56:17.951402 1103 leader_election.cc:288] T 603775196acc416da4295236a90ab22f P 372c38dcee1c43e1af320d7a26511d8c [CANDIDATE]: Term 1 election: Requested vote from peers ac470d791ea04f708719e48abd721021 (127.31.40.1:44729), a8d762c4f42841c186f48a56ecc5354f (127.31.40.2:36135)
I20260928 14:56:17.952041 732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "603775196acc416da4295236a90ab22f" candidate_uuid: "372c38dcee1c43e1af320d7a26511d8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac470d791ea04f708719e48abd721021"
I20260928 14:56:17.952049 878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "603775196acc416da4295236a90ab22f" candidate_uuid: "372c38dcee1c43e1af320d7a26511d8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8d762c4f42841c186f48a56ecc5354f"
I20260928 14:56:17.952255 732 raft_consensus.cc:3037] T 603775196acc416da4295236a90ab22f P ac470d791ea04f708719e48abd721021 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:56:17.952270 878 raft_consensus.cc:3037] T 603775196acc416da4295236a90ab22f P a8d762c4f42841c186f48a56ecc5354f [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:56:17.954591 732 raft_consensus.cc:2454] T 603775196acc416da4295236a90ab22f P ac470d791ea04f708719e48abd721021 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 372c38dcee1c43e1af320d7a26511d8c in term 1.
I20260928 14:56:17.955091 946 leader_election.cc:302] T 603775196acc416da4295236a90ab22f P 372c38dcee1c43e1af320d7a26511d8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 372c38dcee1c43e1af320d7a26511d8c, ac470d791ea04f708719e48abd721021; no voters:
I20260928 14:56:17.955273 878 raft_consensus.cc:2454] T 603775196acc416da4295236a90ab22f P a8d762c4f42841c186f48a56ecc5354f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 372c38dcee1c43e1af320d7a26511d8c in term 1.
I20260928 14:56:17.955407 1103 raft_consensus.cc:2781] T 603775196acc416da4295236a90ab22f P 372c38dcee1c43e1af320d7a26511d8c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:56:17.955602 1103 raft_consensus.cc:686] T 603775196acc416da4295236a90ab22f P 372c38dcee1c43e1af320d7a26511d8c [term 1 LEADER]: Becoming Leader. State: Replica: 372c38dcee1c43e1af320d7a26511d8c, State: Running, Role: LEADER
I20260928 14:56:17.956099 1103 consensus_queue.cc:237] T 603775196acc416da4295236a90ab22f P 372c38dcee1c43e1af320d7a26511d8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac470d791ea04f708719e48abd721021" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 44729 } } peers { permanent_uuid: "a8d762c4f42841c186f48a56ecc5354f" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 36135 } } peers { permanent_uuid: "372c38dcee1c43e1af320d7a26511d8c" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34307 } }
I20260928 14:56:17.958957 573 catalog_manager.cc:5495] T 603775196acc416da4295236a90ab22f P 372c38dcee1c43e1af320d7a26511d8c reported cstate change: term changed from 0 to 1, leader changed from <none> to 372c38dcee1c43e1af320d7a26511d8c (127.31.40.3). New cstate: current_term: 1 leader_uuid: "372c38dcee1c43e1af320d7a26511d8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac470d791ea04f708719e48abd721021" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 44729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8d762c4f42841c186f48a56ecc5354f" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 36135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "372c38dcee1c43e1af320d7a26511d8c" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34307 } health_report { overall_health: HEALTHY } } }
Sep 28 14:56:17 dist-test-slave-j968 krb5kdc[528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607375, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.62@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:56:18.008641 951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:17.994796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57192 (local address 127.31.40.62:34279)
0928 14:56:17.995060 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:17.995065 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:17.995651 (+ 586us) server_negotiation.cc:407] Connection header received
0928 14:56:17.995855 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:17.995860 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:17.995955 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:17.996052 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:17.997256 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:17.998023 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:17.999255 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:17.999677 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:17.999809 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:18.003323 (+ 3514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:18.003329 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:18.003334 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:18.003371 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:18.005993 (+ 2622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:18.006655 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:18.006663 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:18.006666 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:18.006778 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:18.007122 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:18.007129 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:18.007132 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:18.007393 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:18.007545 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:18.007965 (+ 420us) server_negotiation.cc:299] Negotiation successful
0928 14:56:18.008234 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":60}
I20260928 14:56:18.008750 1118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:17.994896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34279 (local address 127.0.0.1:57192)
0928 14:56:17.995536 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:17.995551 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:17.995730 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:17.996088 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:17.996092 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:17.996798 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:56:17.997099 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:17.997109 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:17.998147 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:17.998155 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:17.999121 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:17.999132 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:17.999845 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:17.999851 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:18.000036 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:18.000741 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:56:18.000758 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:56:18.003146 (+ 2388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:56:18.006120 (+ 2974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:18.006135 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:18.006143 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:18.006544 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:18.006875 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:18.006882 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:18.006885 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:18.007024 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:18.007527 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:56:18.007536 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:56:18.007709 (+ 173us) client_negotiation.cc:769] Sending connection context
0928 14:56:18.007868 (+ 159us) client_negotiation.cc:240] Negotiation successful
0928 14:56:18.008348 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"mutex_wait_us":154,"thread_start_us":352,"threads_started":1}
I20260928 14:56:18.014685 573 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:57192:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 14:56:18.015731 573 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:56:18.022342 711 tablet_service.cc:1467] Processing CreateTablet for tablet bea5c0e592a14aaea3afd8c13a295981 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=52cbe8f5aff641c09da67a7e42f0a8b8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:56:18.022632 852 tablet_service.cc:1467] Processing CreateTablet for tablet bea5c0e592a14aaea3afd8c13a295981 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=52cbe8f5aff641c09da67a7e42f0a8b8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:56:18.022852 711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bea5c0e592a14aaea3afd8c13a295981. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:56:18.023157 852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bea5c0e592a14aaea3afd8c13a295981. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:56:18.028487 1083 tablet_bootstrap.cc:492] T bea5c0e592a14aaea3afd8c13a295981 P ac470d791ea04f708719e48abd721021: Bootstrap starting.
I20260928 14:56:18.030172 992 tablet_service.cc:1467] Processing CreateTablet for tablet bea5c0e592a14aaea3afd8c13a295981 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=52cbe8f5aff641c09da67a7e42f0a8b8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:56:18.030900 992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bea5c0e592a14aaea3afd8c13a295981. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:56:18.031697 1083 tablet_bootstrap.cc:654] T bea5c0e592a14aaea3afd8c13a295981 P ac470d791ea04f708719e48abd721021: Neither blocks nor log segments found. Creating new log.
I20260928 14:56:18.032836 1084 tablet_bootstrap.cc:492] T bea5c0e592a14aaea3afd8c13a295981 P a8d762c4f42841c186f48a56ecc5354f: Bootstrap starting.
I20260928 14:56:18.036242 1084 tablet_bootstrap.cc:654] T bea5c0e592a14aaea3afd8c13a295981 P a8d762c4f42841c186f48a56ecc5354f: Neither blocks nor log segments found. Creating new log.
I20260928 14:56:18.041460 1084 tablet_bootstrap.cc:492] T bea5c0e592a14aaea3afd8c13a295981 P a8d762c4f42841c186f48a56ecc5354f: No bootstrap required, opened a new log
I20260928 14:56:18.041664 1084 ts_tablet_manager.cc:1397] T bea5c0e592a14aaea3afd8c13a295981 P a8d762c4f42841c186f48a56ecc5354f: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260928 14:56:18.042274 1085 tablet_bootstrap.cc:492] T bea5c0e592a14aaea3afd8c13a295981 P 372c38dcee1c43e1af320d7a26511d8c: Bootstrap starting.
I20260928 14:56:18.042512 1084 raft_consensus.cc:348] T bea5c0e592a14aaea3afd8c13a295981 P a8d762c4f42841c186f48a56ecc5354f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac470d791ea04f708719e48abd721021" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 44729 } } peers { permanent_uuid: "a8d762c4f42841c186f48a56ecc5354f" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 36135 } } peers { permanent_uuid: "372c38dcee1c43e1af320d7a26511d8c" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34307 } }
I20260928 14:56:18.042776 1084 raft_consensus.cc:374] T bea5c0e592a14aaea3afd8c13a295981 P a8d762c4f42841c186f48a56ecc5354f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:56:18.042850 1084 raft_consensus.cc:729] T bea5c0e592a14aaea3afd8c13a295981 P a8d762c4f42841c186f48a56ecc5354f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8d762c4f42841c186f48a56ecc5354f, State: Initialized, Role: FOLLOWER
I20260928 14:56:18.043157 1084 consensus_queue.cc:260] T bea5c0e592a14aaea3afd8c13a295981 P a8d762c4f42841c186f48a56ecc5354f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac470d791ea04f708719e48abd721021" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 44729 } } peers { permanent_uuid: "a8d762c4f42841c186f48a56ecc5354f" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 36135 } } peers { permanent_uuid: "372c38dcee1c43e1af320d7a26511d8c" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34307 } }
I20260928 14:56:18.043810 1084 ts_tablet_manager.cc:1428] T bea5c0e592a14aaea3afd8c13a295981 P a8d762c4f42841c186f48a56ecc5354f: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 14:56:18.044163 1085 tablet_bootstrap.cc:654] T bea5c0e592a14aaea3afd8c13a295981 P 372c38dcee1c43e1af320d7a26511d8c: Neither blocks nor log segments found. Creating new log.
I20260928 14:56:18.044191 1091 tablet_replica.cc:439] T bea5c0e592a14aaea3afd8c13a295981 P a8d762c4f42841c186f48a56ecc5354f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac470d791ea04f708719e48abd721021" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 44729 } } peers { permanent_uuid: "a8d762c4f42841c186f48a56ecc5354f" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 36135 } } peers { permanent_uuid: "372c38dcee1c43e1af320d7a26511d8c" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34307 } } }
I20260928 14:56:18.044570 1091 tablet_replica.cc:442] T bea5c0e592a14aaea3afd8c13a295981 P a8d762c4f42841c186f48a56ecc5354f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:56:18.047205 1083 tablet_bootstrap.cc:492] T bea5c0e592a14aaea3afd8c13a295981 P ac470d791ea04f708719e48abd721021: No bootstrap required, opened a new log
I20260928 14:56:18.047492 1083 ts_tablet_manager.cc:1397] T bea5c0e592a14aaea3afd8c13a295981 P ac470d791ea04f708719e48abd721021: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.005s
I20260928 14:56:18.050036 1085 tablet_bootstrap.cc:492] T bea5c0e592a14aaea3afd8c13a295981 P 372c38dcee1c43e1af320d7a26511d8c: No bootstrap required, opened a new log
I20260928 14:56:18.050205 1085 ts_tablet_manager.cc:1397] T bea5c0e592a14aaea3afd8c13a295981 P 372c38dcee1c43e1af320d7a26511d8c: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260928 14:56:18.050948 1085 raft_consensus.cc:348] T bea5c0e592a14aaea3afd8c13a295981 P 372c38dcee1c43e1af320d7a26511d8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac470d791ea04f708719e48abd721021" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 44729 } } peers { permanent_uuid: "a8d762c4f42841c186f48a56ecc5354f" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 36135 } } peers { permanent_uuid: "372c38dcee1c43e1af320d7a26511d8c" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34307 } }
I20260928 14:56:18.051194 1085 raft_consensus.cc:374] T bea5c0e592a14aaea3afd8c13a295981 P 372c38dcee1c43e1af320d7a26511d8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:56:18.051263 1085 raft_consensus.cc:729] T bea5c0e592a14aaea3afd8c13a295981 P 372c38dcee1c43e1af320d7a26511d8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 372c38dcee1c43e1af320d7a26511d8c, State: Initialized, Role: FOLLOWER
I20260928 14:56:18.051539 1085 consensus_queue.cc:260] T bea5c0e592a14aaea3afd8c13a295981 P 372c38dcee1c43e1af320d7a26511d8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac470d791ea04f708719e48abd721021" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 44729 } } peers { permanent_uuid: "a8d762c4f42841c186f48a56ecc5354f" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 36135 } } peers { permanent_uuid: "372c38dcee1c43e1af320d7a26511d8c" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34307 } }
I20260928 14:56:18.052557 1083 raft_consensus.cc:348] T bea5c0e592a14aaea3afd8c13a295981 P ac470d791ea04f708719e48abd721021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac470d791ea04f708719e48abd721021" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 44729 } } peers { permanent_uuid: "a8d762c4f42841c186f48a56ecc5354f" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 36135 } } peers { permanent_uuid: "372c38dcee1c43e1af320d7a26511d8c" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34307 } }
I20260928 14:56:18.052841 1083 raft_consensus.cc:374] T bea5c0e592a14aaea3afd8c13a295981 P ac470d791ea04f708719e48abd721021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:56:18.052919 1083 raft_consensus.cc:729] T bea5c0e592a14aaea3afd8c13a295981 P ac470d791ea04f708719e48abd721021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac470d791ea04f708719e48abd721021, State: Initialized, Role: FOLLOWER
I20260928 14:56:18.052794 1103 tablet_replica.cc:439] T bea5c0e592a14aaea3afd8c13a295981 P 372c38dcee1c43e1af320d7a26511d8c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac470d791ea04f708719e48abd721021" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 44729 } } peers { permanent_uuid: "a8d762c4f42841c186f48a56ecc5354f" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 36135 } } peers { permanent_uuid: "372c38dcee1c43e1af320d7a26511d8c" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34307 } } }
I20260928 14:56:18.053148 1103 tablet_replica.cc:442] T bea5c0e592a14aaea3afd8c13a295981 P 372c38dcee1c43e1af320d7a26511d8c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:56:18.053221 1083 consensus_queue.cc:260] T bea5c0e592a14aaea3afd8c13a295981 P ac470d791ea04f708719e48abd721021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac470d791ea04f708719e48abd721021" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 44729 } } peers { permanent_uuid: "a8d762c4f42841c186f48a56ecc5354f" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 36135 } } peers { permanent_uuid: "372c38dcee1c43e1af320d7a26511d8c" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34307 } }
I20260928 14:56:18.053654 1085 ts_tablet_manager.cc:1428] T bea5c0e592a14aaea3afd8c13a295981 P 372c38dcee1c43e1af320d7a26511d8c: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 14:56:18.053826 1083 ts_tablet_manager.cc:1428] T bea5c0e592a14aaea3afd8c13a295981 P ac470d791ea04f708719e48abd721021: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20260928 14:56:18.053726 1090 tablet_replica.cc:439] T bea5c0e592a14aaea3afd8c13a295981 P ac470d791ea04f708719e48abd721021: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac470d791ea04f708719e48abd721021" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 44729 } } peers { permanent_uuid: "a8d762c4f42841c186f48a56ecc5354f" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 36135 } } peers { permanent_uuid: "372c38dcee1c43e1af320d7a26511d8c" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34307 } } }
I20260928 14:56:18.054561 1090 tablet_replica.cc:442] T bea5c0e592a14aaea3afd8c13a295981 P ac470d791ea04f708719e48abd721021: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:56:18.324365 1090 raft_consensus.cc:482] T bea5c0e592a14aaea3afd8c13a295981 P ac470d791ea04f708719e48abd721021 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:56:18.324622 1090 raft_consensus.cc:504] T bea5c0e592a14aaea3afd8c13a295981 P ac470d791ea04f708719e48abd721021 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac470d791ea04f708719e48abd721021" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 44729 } } peers { permanent_uuid: "a8d762c4f42841c186f48a56ecc5354f" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 36135 } } peers { permanent_uuid: "372c38dcee1c43e1af320d7a26511d8c" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34307 } }
I20260928 14:56:18.326198 1090 leader_election.cc:288] T bea5c0e592a14aaea3afd8c13a295981 P ac470d791ea04f708719e48abd721021 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8d762c4f42841c186f48a56ecc5354f (127.31.40.2:36135), 372c38dcee1c43e1af320d7a26511d8c (127.31.40.3:34307)
I20260928 14:56:18.348906 1124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:18.326633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:40189 (local address 127.31.40.2:36135)
0928 14:56:18.340056 (+ 13423us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:18.340062 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:18.340083 (+ 21us) server_negotiation.cc:407] Connection header received
0928 14:56:18.340175 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:18.340180 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:18.340272 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:18.340365 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:56:18.341896 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:18.343086 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:18.345817 (+ 2731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:18.347006 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:18.347136 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:18.347248 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:18.348412 (+ 1164us) server_negotiation.cc:299] Negotiation successful
0928 14:56:18.348576 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13297,"thread_start_us":13179,"threads_started":1}
I20260928 14:56:18.349614 1123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:18.326496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.2:36135 (local address 127.31.40.1:40189)
0928 14:56:18.339186 (+ 12690us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:18.339209 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:18.339376 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:18.341068 (+ 1692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:18.341073 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:18.341113 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:56:18.341401 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:18.341411 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:18.343260 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:18.343266 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:18.345631 (+ 2365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:18.345646 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:18.347357 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:18.347363 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:18.347551 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:18.347584 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 14:56:18.347670 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 14:56:18.347807 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12535,"thread_start_us":21381,"threads_started":1}
I20260928 14:56:18.350494 878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bea5c0e592a14aaea3afd8c13a295981" candidate_uuid: "ac470d791ea04f708719e48abd721021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8d762c4f42841c186f48a56ecc5354f" is_pre_election: true
I20260928 14:56:18.350772 878 raft_consensus.cc:2454] T bea5c0e592a14aaea3afd8c13a295981 P a8d762c4f42841c186f48a56ecc5354f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac470d791ea04f708719e48abd721021 in term 0.
I20260928 14:56:18.351326 664 leader_election.cc:302] T bea5c0e592a14aaea3afd8c13a295981 P ac470d791ea04f708719e48abd721021 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8d762c4f42841c186f48a56ecc5354f, ac470d791ea04f708719e48abd721021; no voters:
I20260928 14:56:18.351814 1090 raft_consensus.cc:2781] T bea5c0e592a14aaea3afd8c13a295981 P ac470d791ea04f708719e48abd721021 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:56:18.351982 1090 raft_consensus.cc:482] T bea5c0e592a14aaea3afd8c13a295981 P ac470d791ea04f708719e48abd721021 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:56:18.352103 1090 raft_consensus.cc:3037] T bea5c0e592a14aaea3afd8c13a295981 P ac470d791ea04f708719e48abd721021 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:56:18.355512 1090 raft_consensus.cc:504] T bea5c0e592a14aaea3afd8c13a295981 P ac470d791ea04f708719e48abd721021 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac470d791ea04f708719e48abd721021" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 44729 } } peers { permanent_uuid: "a8d762c4f42841c186f48a56ecc5354f" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 36135 } } peers { permanent_uuid: "372c38dcee1c43e1af320d7a26511d8c" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34307 } }
I20260928 14:56:18.356482 1090 leader_election.cc:288] T bea5c0e592a14aaea3afd8c13a295981 P ac470d791ea04f708719e48abd721021 [CANDIDATE]: Term 1 election: Requested vote from peers a8d762c4f42841c186f48a56ecc5354f (127.31.40.2:36135), 372c38dcee1c43e1af320d7a26511d8c (127.31.40.3:34307)
I20260928 14:56:18.356834 1125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:18.327077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:52557 (local address 127.31.40.3:34307)
0928 14:56:18.340976 (+ 13899us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:18.340983 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:18.348563 (+ 7580us) server_negotiation.cc:407] Connection header received
0928 14:56:18.348736 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:18.348742 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:18.348836 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:18.348982 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:56:18.349852 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:18.351141 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:18.354097 (+ 2956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:18.355422 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:18.355538 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:18.355645 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:18.356257 (+ 612us) server_negotiation.cc:299] Negotiation successful
0928 14:56:18.356446 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13771,"thread_start_us":12446,"threads_started":1}
I20260928 14:56:18.357118 878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bea5c0e592a14aaea3afd8c13a295981" candidate_uuid: "ac470d791ea04f708719e48abd721021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8d762c4f42841c186f48a56ecc5354f"
I20260928 14:56:18.357391 878 raft_consensus.cc:3037] T bea5c0e592a14aaea3afd8c13a295981 P a8d762c4f42841c186f48a56ecc5354f [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:56:18.357774 1122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:18.326531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.3:34307 (local address 127.31.40.1:52557)
0928 14:56:18.348427 (+ 21896us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:18.348447 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:18.348578 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:18.349171 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:18.349178 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:18.349210 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:56:18.349651 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:18.349667 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:18.351337 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:18.351343 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:18.353839 (+ 2496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:18.353858 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:18.355846 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:18.355853 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:18.356052 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:18.356085 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 14:56:18.357117 (+ 1032us) client_negotiation.cc:240] Negotiation successful
0928 14:56:18.357289 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21783,"mutex_wait_us":4,"thread_start_us":21624,"threads_started":1}
I20260928 14:56:18.359076 1014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bea5c0e592a14aaea3afd8c13a295981" candidate_uuid: "ac470d791ea04f708719e48abd721021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "372c38dcee1c43e1af320d7a26511d8c" is_pre_election: true
I20260928 14:56:18.359402 1013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bea5c0e592a14aaea3afd8c13a295981" candidate_uuid: "ac470d791ea04f708719e48abd721021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "372c38dcee1c43e1af320d7a26511d8c"
I20260928 14:56:18.359675 1014 raft_consensus.cc:2454] T bea5c0e592a14aaea3afd8c13a295981 P 372c38dcee1c43e1af320d7a26511d8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac470d791ea04f708719e48abd721021 in term 0.
I20260928 14:56:18.360363 878 raft_consensus.cc:2454] T bea5c0e592a14aaea3afd8c13a295981 P a8d762c4f42841c186f48a56ecc5354f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac470d791ea04f708719e48abd721021 in term 1.
I20260928 14:56:18.360862 664 leader_election.cc:302] T bea5c0e592a14aaea3afd8c13a295981 P ac470d791ea04f708719e48abd721021 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8d762c4f42841c186f48a56ecc5354f, ac470d791ea04f708719e48abd721021; no voters:
I20260928 14:56:18.361202 1090 raft_consensus.cc:2781] T bea5c0e592a14aaea3afd8c13a295981 P ac470d791ea04f708719e48abd721021 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:56:18.361685 1090 raft_consensus.cc:686] T bea5c0e592a14aaea3afd8c13a295981 P ac470d791ea04f708719e48abd721021 [term 1 LEADER]: Becoming Leader. State: Replica: ac470d791ea04f708719e48abd721021, State: Running, Role: LEADER
I20260928 14:56:18.362208 1090 consensus_queue.cc:237] T bea5c0e592a14aaea3afd8c13a295981 P ac470d791ea04f708719e48abd721021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac470d791ea04f708719e48abd721021" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 44729 } } peers { permanent_uuid: "a8d762c4f42841c186f48a56ecc5354f" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 36135 } } peers { permanent_uuid: "372c38dcee1c43e1af320d7a26511d8c" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34307 } }
I20260928 14:56:18.365615 1126 tablet_replica.cc:439] T bea5c0e592a14aaea3afd8c13a295981 P ac470d791ea04f708719e48abd721021: TxnStatusTablet state changed. Reason: New leader ac470d791ea04f708719e48abd721021. Latest consensus state: current_term: 1 leader_uuid: "ac470d791ea04f708719e48abd721021" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac470d791ea04f708719e48abd721021" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 44729 } } peers { permanent_uuid: "a8d762c4f42841c186f48a56ecc5354f" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 36135 } } peers { permanent_uuid: "372c38dcee1c43e1af320d7a26511d8c" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34307 } } }
I20260928 14:56:18.366092 1126 tablet_replica.cc:442] T bea5c0e592a14aaea3afd8c13a295981 P ac470d791ea04f708719e48abd721021: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:56:18.366910 1128 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 14:56:18.367180 1128 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 14:56:18.367756 573 catalog_manager.cc:5495] T bea5c0e592a14aaea3afd8c13a295981 P ac470d791ea04f708719e48abd721021 reported cstate change: term changed from 0 to 1, leader changed from <none> to ac470d791ea04f708719e48abd721021 (127.31.40.1). New cstate: current_term: 1 leader_uuid: "ac470d791ea04f708719e48abd721021" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac470d791ea04f708719e48abd721021" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 44729 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8d762c4f42841c186f48a56ecc5354f" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 36135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "372c38dcee1c43e1af320d7a26511d8c" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34307 } health_report { overall_health: UNKNOWN } } }
I20260928 14:56:18.429623 1131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:18.421605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41594 (local address 127.31.40.1:44729)
0928 14:56:18.422424 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:18.422430 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:18.422451 (+ 21us) server_negotiation.cc:407] Connection header received
0928 14:56:18.422551 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:18.422558 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:18.422689 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:18.422861 (+ 172us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:56:18.423327 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:18.424690 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:18.426448 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:18.427278 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:18.427480 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:18.428025 (+ 545us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:56:18.428849 (+ 824us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:56:18.429015 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:18.429131 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 14:56:18.429229 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":455,"threads_started":1}
I20260928 14:56:18.429642 1118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:18.421380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:44729 (local address 127.0.0.1:41594)
0928 14:56:18.421646 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:18.421670 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:18.421838 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:18.422906 (+ 1068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:18.422910 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:18.422923 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:56:18.423160 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:18.423169 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:18.424819 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:18.424823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:18.426293 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:18.426305 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:18.427464 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:18.427469 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:18.427686 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:18.427743 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:56:18.428984 (+ 1241us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:56:18.429003 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 14:56:18.429089 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 14:56:18.429275 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 14:56:18.436342 1013 raft_consensus.cc:3037] T bea5c0e592a14aaea3afd8c13a295981 P 372c38dcee1c43e1af320d7a26511d8c [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:56:18.436930 878 raft_consensus.cc:1261] T bea5c0e592a14aaea3afd8c13a295981 P a8d762c4f42841c186f48a56ecc5354f [term 1 FOLLOWER]: Refusing update from remote peer ac470d791ea04f708719e48abd721021: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:56:18.437666 1126 consensus_queue.cc:1035] T bea5c0e592a14aaea3afd8c13a295981 P ac470d791ea04f708719e48abd721021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8d762c4f42841c186f48a56ecc5354f" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 36135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:56:18.440706 1013 raft_consensus.cc:1261] T bea5c0e592a14aaea3afd8c13a295981 P 372c38dcee1c43e1af320d7a26511d8c [term 1 FOLLOWER]: Refusing update from remote peer ac470d791ea04f708719e48abd721021: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:56:18.441740 1126 consensus_queue.cc:1035] T bea5c0e592a14aaea3afd8c13a295981 P ac470d791ea04f708719e48abd721021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "372c38dcee1c43e1af320d7a26511d8c" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:56:18.453734 1103 tablet_replica.cc:439] T bea5c0e592a14aaea3afd8c13a295981 P 372c38dcee1c43e1af320d7a26511d8c: TxnStatusTablet state changed. Reason: New leader ac470d791ea04f708719e48abd721021. Latest consensus state: current_term: 1 leader_uuid: "ac470d791ea04f708719e48abd721021" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac470d791ea04f708719e48abd721021" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 44729 } } peers { permanent_uuid: "a8d762c4f42841c186f48a56ecc5354f" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 36135 } } peers { permanent_uuid: "372c38dcee1c43e1af320d7a26511d8c" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34307 } } }
I20260928 14:56:18.454356 1103 tablet_replica.cc:442] T bea5c0e592a14aaea3afd8c13a295981 P 372c38dcee1c43e1af320d7a26511d8c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:56:18.462636 1091 tablet_replica.cc:439] T bea5c0e592a14aaea3afd8c13a295981 P a8d762c4f42841c186f48a56ecc5354f: TxnStatusTablet state changed. Reason: New leader ac470d791ea04f708719e48abd721021. Latest consensus state: current_term: 1 leader_uuid: "ac470d791ea04f708719e48abd721021" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac470d791ea04f708719e48abd721021" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 44729 } } peers { permanent_uuid: "a8d762c4f42841c186f48a56ecc5354f" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 36135 } } peers { permanent_uuid: "372c38dcee1c43e1af320d7a26511d8c" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34307 } } }
I20260928 14:56:18.463109 1091 tablet_replica.cc:442] T bea5c0e592a14aaea3afd8c13a295981 P a8d762c4f42841c186f48a56ecc5354f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:56:18.466488 1103 tablet_replica.cc:439] T bea5c0e592a14aaea3afd8c13a295981 P 372c38dcee1c43e1af320d7a26511d8c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ac470d791ea04f708719e48abd721021" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac470d791ea04f708719e48abd721021" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 44729 } } peers { permanent_uuid: "a8d762c4f42841c186f48a56ecc5354f" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 36135 } } peers { permanent_uuid: "372c38dcee1c43e1af320d7a26511d8c" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34307 } } }
I20260928 14:56:18.466778 1103 tablet_replica.cc:442] T bea5c0e592a14aaea3afd8c13a295981 P 372c38dcee1c43e1af320d7a26511d8c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:56:18.470855 1139 consensus_queue.cc:1035] T 603775196acc416da4295236a90ab22f P 372c38dcee1c43e1af320d7a26511d8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8d762c4f42841c186f48a56ecc5354f" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 36135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260928 14:56:18.478153 1090 tablet_replica.cc:439] T bea5c0e592a14aaea3afd8c13a295981 P ac470d791ea04f708719e48abd721021: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ac470d791ea04f708719e48abd721021" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac470d791ea04f708719e48abd721021" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 44729 } } peers { permanent_uuid: "a8d762c4f42841c186f48a56ecc5354f" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 36135 } } peers { permanent_uuid: "372c38dcee1c43e1af320d7a26511d8c" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34307 } } }
I20260928 14:56:18.478641 1090 tablet_replica.cc:442] T bea5c0e592a14aaea3afd8c13a295981 P ac470d791ea04f708719e48abd721021: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:56:18.479655 1090 tablet_replica.cc:439] T bea5c0e592a14aaea3afd8c13a295981 P ac470d791ea04f708719e48abd721021: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ac470d791ea04f708719e48abd721021" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac470d791ea04f708719e48abd721021" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 44729 } } peers { permanent_uuid: "a8d762c4f42841c186f48a56ecc5354f" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 36135 } } peers { permanent_uuid: "372c38dcee1c43e1af320d7a26511d8c" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34307 } } }
I20260928 14:56:18.479961 1090 tablet_replica.cc:442] T bea5c0e592a14aaea3afd8c13a295981 P ac470d791ea04f708719e48abd721021: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:56:18.491667 1091 tablet_replica.cc:439] T bea5c0e592a14aaea3afd8c13a295981 P a8d762c4f42841c186f48a56ecc5354f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ac470d791ea04f708719e48abd721021" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac470d791ea04f708719e48abd721021" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 44729 } } peers { permanent_uuid: "a8d762c4f42841c186f48a56ecc5354f" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 36135 } } peers { permanent_uuid: "372c38dcee1c43e1af320d7a26511d8c" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34307 } } }
I20260928 14:56:18.492002 1091 tablet_replica.cc:442] T bea5c0e592a14aaea3afd8c13a295981 P a8d762c4f42841c186f48a56ecc5354f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:56:18.507189 1103 consensus_queue.cc:1035] T 603775196acc416da4295236a90ab22f P 372c38dcee1c43e1af320d7a26511d8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac470d791ea04f708719e48abd721021" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 44729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:56:18.527251 1131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:18.521360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:38927 (local address 127.31.40.1:44729)
0928 14:56:18.521549 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:18.521557 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:18.522290 (+ 733us) server_negotiation.cc:407] Connection header received
0928 14:56:18.522503 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:18.522509 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:18.522584 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:18.522675 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:56:18.523167 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:18.524020 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:18.525358 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:18.525856 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:18.526016 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:18.526452 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:56:18.526565 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:56:18.526658 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:18.526795 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 14:56:18.526877 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 14:56:18.527310 1147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:18.521382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:44729 (local address 127.31.40.3:38927)
0928 14:56:18.522181 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:18.522198 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:18.522369 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:18.522694 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:18.522698 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:18.522710 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:56:18.523018 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:18.523027 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:18.524158 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:18.524163 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:18.525216 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:18.525229 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:18.526055 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:18.526063 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:18.526202 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:18.526261 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:56:18.526660 (+ 399us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:56:18.526674 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 14:56:18.526768 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 14:56:18.526919 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":378,"threads_started":1}
I20260928 14:56:18.570122 1125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:18.564041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:35691 (local address 127.31.40.3:34307)
0928 14:56:18.564242 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:18.564251 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:18.564802 (+ 551us) server_negotiation.cc:407] Connection header received
0928 14:56:18.565017 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:18.565022 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:18.565093 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:18.565184 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:56:18.565652 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:18.566646 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:18.568266 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:18.568740 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:18.568862 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:18.569441 (+ 579us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:56:18.569550 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:56:18.569642 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:18.569758 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 14:56:18.569838 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 14:56:18.570248 1150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:18.563990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.3:34307 (local address 127.31.40.1:35691)
0928 14:56:18.564688 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:18.564712 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:18.564882 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:18.565203 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:18.565208 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:18.565219 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:56:18.565494 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:18.565506 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:18.566780 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:18.566785 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:18.568114 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:18.568126 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:18.568990 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:18.568994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:18.569175 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:18.569253 (+ 78us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:56:18.569643 (+ 390us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:56:18.569657 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 14:56:18.569734 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 14:56:18.569881 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":306,"threads_started":1}
I20260928 14:56:18.613636 573 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:56:18.613873 573 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:56:18.616892 573 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 633f7fb706e243119cb41b0ad22c5be6: Sending DeleteTablet for 3 replicas of tablet 603775196acc416da4295236a90ab22f
I20260928 14:56:18.618299 992 tablet_service.cc:1514] Processing DeleteTablet for tablet 603775196acc416da4295236a90ab22f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:56:18 UTC) from {username='kudu', principal='kudu/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:39810
I20260928 14:56:18.618641 852 tablet_service.cc:1514] Processing DeleteTablet for tablet 603775196acc416da4295236a90ab22f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:56:18 UTC) from {username='kudu', principal='kudu/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:48030
I20260928 14:56:18.619130 1152 tablet_replica.cc:330] T 603775196acc416da4295236a90ab22f P 372c38dcee1c43e1af320d7a26511d8c: stopping tablet replica
I20260928 14:56:18.619356 1153 tablet_replica.cc:330] T 603775196acc416da4295236a90ab22f P a8d762c4f42841c186f48a56ecc5354f: stopping tablet replica
I20260928 14:56:18.619364 1152 raft_consensus.cc:2229] T 603775196acc416da4295236a90ab22f P 372c38dcee1c43e1af320d7a26511d8c [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:56:18.619620 1153 raft_consensus.cc:2229] T 603775196acc416da4295236a90ab22f P a8d762c4f42841c186f48a56ecc5354f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:56:18.619765 1152 raft_consensus.cc:2258] T 603775196acc416da4295236a90ab22f P 372c38dcee1c43e1af320d7a26511d8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:56:18.620056 1153 raft_consensus.cc:2258] T 603775196acc416da4295236a90ab22f P a8d762c4f42841c186f48a56ecc5354f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:56:18.620975 1152 ts_tablet_manager.cc:1905] T 603775196acc416da4295236a90ab22f P 372c38dcee1c43e1af320d7a26511d8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:56:18.621927 711 tablet_service.cc:1514] Processing DeleteTablet for tablet 603775196acc416da4295236a90ab22f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:56:18 UTC) from {username='kudu', principal='kudu/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:59954
I20260928 14:56:18.627545 1153 ts_tablet_manager.cc:1905] T 603775196acc416da4295236a90ab22f P a8d762c4f42841c186f48a56ecc5354f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:56:18.628379 1155 tablet_replica.cc:330] T 603775196acc416da4295236a90ab22f P ac470d791ea04f708719e48abd721021: stopping tablet replica
I20260928 14:56:18.628618 1155 raft_consensus.cc:2229] T 603775196acc416da4295236a90ab22f P ac470d791ea04f708719e48abd721021 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:56:18.628923 1155 raft_consensus.cc:2258] T 603775196acc416da4295236a90ab22f P ac470d791ea04f708719e48abd721021 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:56:18.630100 1155 ts_tablet_manager.cc:1905] T 603775196acc416da4295236a90ab22f P ac470d791ea04f708719e48abd721021: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:56:18.634318 1152 ts_tablet_manager.cc:1918] T 603775196acc416da4295236a90ab22f P 372c38dcee1c43e1af320d7a26511d8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:56:18.634461 1152 log.cc:1196] T 603775196acc416da4295236a90ab22f P 372c38dcee1c43e1af320d7a26511d8c: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/ts-2/wal/wals/603775196acc416da4295236a90ab22f
I20260928 14:56:18.634955 1152 ts_tablet_manager.cc:1939] T 603775196acc416da4295236a90ab22f P 372c38dcee1c43e1af320d7a26511d8c: Deleting consensus metadata
I20260928 14:56:18.635025 1153 ts_tablet_manager.cc:1918] T 603775196acc416da4295236a90ab22f P a8d762c4f42841c186f48a56ecc5354f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:56:18.635172 1153 log.cc:1196] T 603775196acc416da4295236a90ab22f P a8d762c4f42841c186f48a56ecc5354f: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/ts-1/wal/wals/603775196acc416da4295236a90ab22f
I20260928 14:56:18.635630 1153 ts_tablet_manager.cc:1939] T 603775196acc416da4295236a90ab22f P a8d762c4f42841c186f48a56ecc5354f: Deleting consensus metadata
I20260928 14:56:18.636438 559 catalog_manager.cc:4841] TS 372c38dcee1c43e1af320d7a26511d8c (127.31.40.3:34307): tablet 603775196acc416da4295236a90ab22f (table test-table [id=1098dd7f638d472ab27285a340a53950]) successfully deleted
I20260928 14:56:18.636734 561 catalog_manager.cc:4841] TS a8d762c4f42841c186f48a56ecc5354f (127.31.40.2:36135): tablet 603775196acc416da4295236a90ab22f (table test-table [id=1098dd7f638d472ab27285a340a53950]) successfully deleted
I20260928 14:56:18.640937 1155 ts_tablet_manager.cc:1918] T 603775196acc416da4295236a90ab22f P ac470d791ea04f708719e48abd721021: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:56:18.641124 1155 log.cc:1196] T 603775196acc416da4295236a90ab22f P ac470d791ea04f708719e48abd721021: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607365362981-31904-0/minicluster-data/ts-0/wal/wals/603775196acc416da4295236a90ab22f
I20260928 14:56:18.641687 1155 ts_tablet_manager.cc:1939] T 603775196acc416da4295236a90ab22f P ac470d791ea04f708719e48abd721021: Deleting consensus metadata
I20260928 14:56:18.643151 561 catalog_manager.cc:4841] TS ac470d791ea04f708719e48abd721021 (127.31.40.1:44729): tablet 603775196acc416da4295236a90ab22f (table test-table [id=1098dd7f638d472ab27285a340a53950]) successfully deleted
Sep 28 14:56:18 dist-test-slave-j968 krb5kdc[528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607377, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.40.1@KRBTEST.COM
I20260928 14:56:18.656785 1131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:18.622107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41610 (local address 127.31.40.1:44729)
0928 14:56:18.622325 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:18.622331 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:18.640743 (+ 18412us) server_negotiation.cc:407] Connection header received
0928 14:56:18.640987 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:18.640993 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:18.641077 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:18.641190 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:18.643169 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:18.644202 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:18.645498 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:18.645961 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:18.646113 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:18.650425 (+ 4312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:18.650439 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:18.650456 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:18.650493 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:18.653765 (+ 3272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:18.654498 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:18.654511 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:18.654513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:18.654663 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:18.655041 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:18.655047 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:18.655050 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:18.655428 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:18.655752 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:18.656107 (+ 355us) server_negotiation.cc:299] Negotiation successful
0928 14:56:18.656357 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
W20260928 14:56:18.657642 777 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41610
I20260928 14:56:18.683472 1159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:18.661993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36202 (local address 127.31.40.62:34279)
0928 14:56:18.665955 (+ 3962us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:18.665961 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:18.666688 (+ 727us) server_negotiation.cc:407] Connection header received
0928 14:56:18.666880 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:18.666886 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:18.666977 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:18.667091 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:18.668835 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:18.670048 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:18.672066 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:18.672769 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:18.672900 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:18.676290 (+ 3390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:18.676297 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:18.676303 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:18.676359 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:18.680297 (+ 3938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:18.681293 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:18.681299 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:18.681303 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:18.681461 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:18.681879 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:18.681884 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:18.681887 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:18.682265 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:18.682502 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:18.682847 (+ 345us) server_negotiation.cc:299] Negotiation successful
0928 14:56:18.683101 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3807,"thread_start_us":3630,"threads_started":1}
W20260928 14:56:18.684042 573 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36202
I20260928 14:56:18.685993 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 645
I20260928 14:56:18.705258 1131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:18.687735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41612 (local address 127.31.40.1:44729)
0928 14:56:18.688016 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:18.688023 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:18.691287 (+ 3264us) server_negotiation.cc:407] Connection header received
0928 14:56:18.691492 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:18.691498 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:18.691578 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:18.691690 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:18.692895 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:18.693601 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:18.694859 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:18.695344 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:18.695494 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:18.698042 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:18.698049 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:18.698055 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:18.698101 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:18.702077 (+ 3976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:18.702946 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:18.702952 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:18.702956 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:18.703117 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:18.703471 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:18.703477 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:18.703480 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:18.703918 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:18.704188 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:18.704478 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 14:56:18.704753 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260928 14:56:18.706274 777 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41612
W20260928 14:56:18.706938 31904 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 645: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 14:56:18.707156 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 645
I20260928 14:56:18.727943 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 786
Sep 28 14:56:18 dist-test-slave-j968 krb5kdc[528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607377, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.40.2@KRBTEST.COM
I20260928 14:56:18.744294 1124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:18.728769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38626 (local address 127.31.40.2:36135)
0928 14:56:18.728961 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:18.728967 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:18.728994 (+ 27us) server_negotiation.cc:407] Connection header received
0928 14:56:18.729061 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:18.729065 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:18.729129 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:18.729238 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:18.730390 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:18.731217 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:18.732616 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:18.733091 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:18.733212 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:18.737926 (+ 4714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:18.737944 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:18.737962 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:18.738006 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:18.741287 (+ 3281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:18.742097 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:18.742106 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:18.742108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:18.742281 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:18.742641 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:18.742644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:18.742647 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:18.743021 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:18.743356 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:18.743631 (+ 275us) server_negotiation.cc:299] Negotiation successful
0928 14:56:18.743836 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260928 14:56:18.744980 918 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38626
W20260928 14:56:18.745621 31904 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 786: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 14:56:18.745779 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 786
I20260928 14:56:18.765157 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 927
Sep 28 14:56:18 dist-test-slave-j968 krb5kdc[528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607377, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.40.3@KRBTEST.COM
I20260928 14:56:18.782308 1125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:18.765945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34976 (local address 127.31.40.3:34307)
0928 14:56:18.766141 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:18.766145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:18.766210 (+ 65us) server_negotiation.cc:407] Connection header received
0928 14:56:18.766387 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:18.766391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:18.766463 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:18.766551 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:18.768047 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:18.768938 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:18.770294 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:18.770721 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:18.770830 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:18.775796 (+ 4966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:18.775814 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:18.775832 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:18.775932 (+ 100us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:18.779432 (+ 3500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:18.780311 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:18.780320 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:18.780322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:18.780474 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:18.780830 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:18.780834 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:18.780837 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:18.781252 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:18.781488 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:18.781752 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 14:56:18.781955 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260928 14:56:18.783038 1059 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34976
W20260928 14:56:18.783725 31904 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 927: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 14:56:18.784024 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 927
I20260928 14:56:18.805871 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 541
I20260928 14:56:18.807147 614 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:56:18.922098 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 541
2026-09-28T14:56:18Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5064 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:56:19 dist-test-slave-j968 krb5kdc[1165](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:56:19 dist-test-slave-j968 krb5kdc[1165](info): set up 2 sockets
Sep 28 14:56:19 dist-test-slave-j968 krb5kdc[1165](info): commencing operation
krb5kdc: starting...
W20260928 14:56:20.006441 31904 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Sep 28 14:56:20 dist-test-slave-j968 krb5kdc[1165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:56:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:56:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab.
Entry for principal kudu/127.31.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab.
I20260928 14:56:20.202289 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:43811
--webserver_interface=127.31.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:38353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.40.62:43811
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:56:20.526001 1179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:56:20.526335 1179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:56:20.526427 1179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:56:20.533919 1179 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:56:20.534147 1179 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 14:56:20.534240 1179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:56:20.534305 1179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:56:20.534363 1179 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:56:20.534422 1179 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:56:20.545598 1179 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:38353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.40.62:43811
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:43811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:56:20.546936 1179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:56:20.548246 1179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:56:20.561070 1185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:20.561008 1184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:20.561137 1187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:20.561841 1179 server_base.cc:1034] running on GCE node
I20260928 14:56:20.563081 1179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:56:20.564896 1179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:56:20.566184 1179 hybrid_clock.cc:648] HybridClock initialized: now 1790607380566099 us; error 77 us; skew 500 ppm
I20260928 14:56:20.566584 1179 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:56:20 dist-test-slave-j968 krb5kdc[1165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607380, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:56:20.569988 1179 init.cc:377] Logged in from keytab as kudu/127.31.40.62@KRBTEST.COM (short username kudu)
I20260928 14:56:20.571587 1179 webserver.cc:460] Webserver started at http://127.31.40.62:41523/ using document root <none> and password file <none>
I20260928 14:56:20.572397 1179 fs_manager.cc:359] Metadata directory not provided
I20260928 14:56:20.572527 1179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:56:20.572844 1179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:56:20.575981 1179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/master-0/data/instance:
uuid: "b9e26c8ffca249b9a82a4ec88f68f23b"
format_stamp: "Formatted at 2026-09-28 14:56:20 on dist-test-slave-j968"
server_key: "2f1d9bfec1b0a9657cf24121b698db41"
server_key_iv: "fe0f5e02638d0545c9728e18255a5d9b"
server_key_version: "encryptionkey@0"
I20260928 14:56:20.576799 1179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/master-0/wal/instance:
uuid: "b9e26c8ffca249b9a82a4ec88f68f23b"
format_stamp: "Formatted at 2026-09-28 14:56:20 on dist-test-slave-j968"
server_key: "2f1d9bfec1b0a9657cf24121b698db41"
server_key_iv: "fe0f5e02638d0545c9728e18255a5d9b"
server_key_version: "encryptionkey@0"
I20260928 14:56:20.582333 1179 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 14:56:20.586012 1194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:20.587502 1179 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260928 14:56:20.587693 1179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/master-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/master-0/wal
uuid: "b9e26c8ffca249b9a82a4ec88f68f23b"
format_stamp: "Formatted at 2026-09-28 14:56:20 on dist-test-slave-j968"
server_key: "2f1d9bfec1b0a9657cf24121b698db41"
server_key_iv: "fe0f5e02638d0545c9728e18255a5d9b"
server_key_version: "encryptionkey@0"
I20260928 14:56:20.587936 1179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:56:20.620370 1179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:56:20.625181 1179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:56:20.625538 1179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:56:20.648180 1179 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.62:43811
I20260928 14:56:20.648161 1256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.62:43811 every 8 connection(s)
I20260928 14:56:20.649667 1179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
I20260928 14:56:20.651142 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 1179
I20260928 14:56:20.651345 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/master-0/wal/instance
I20260928 14:56:20.652249 31904 external_mini_cluster.cc:1428] Setting key 0537b1d4eb9a834f56d86b0b9cb2f16b
I20260928 14:56:20.654819 1257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 14:56:20 dist-test-slave-j968 krb5kdc[1165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:56:20.669093 1257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9e26c8ffca249b9a82a4ec88f68f23b: Bootstrap starting.
I20260928 14:56:20.673658 1257 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9e26c8ffca249b9a82a4ec88f68f23b: Neither blocks nor log segments found. Creating new log.
I20260928 14:56:20.675153 1257 log.cc:824] T 00000000000000000000000000000000 P b9e26c8ffca249b9a82a4ec88f68f23b: Log is configured to *not* fsync() on all Append() calls
I20260928 14:56:20.678669 1260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:20.654385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43816 (local address 127.31.40.62:43811)
0928 14:56:20.655211 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:20.655224 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:20.655266 (+ 42us) server_negotiation.cc:407] Connection header received
0928 14:56:20.656966 (+ 1700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:20.656989 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:20.657425 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:20.657796 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:20.659677 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:20.661143 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:20.662959 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:20.663648 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:20.663792 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:20.668929 (+ 5137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:20.668941 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:20.668963 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:20.669017 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:20.673122 (+ 4105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:20.674121 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:20.674136 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:20.674147 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:20.674403 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:20.674937 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:20.674942 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:20.674946 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:20.675797 (+ 851us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:20.676131 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:20.676706 (+ 575us) server_negotiation.cc:299] Negotiation successful
0928 14:56:20.677221 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":392,"threads_started":1}
I20260928 14:56:20.679255 1257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9e26c8ffca249b9a82a4ec88f68f23b: No bootstrap required, opened a new log
I20260928 14:56:20.687397 1257 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9e26c8ffca249b9a82a4ec88f68f23b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9e26c8ffca249b9a82a4ec88f68f23b" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 43811 } }
I20260928 14:56:20.687857 1257 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9e26c8ffca249b9a82a4ec88f68f23b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:56:20.688006 1257 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9e26c8ffca249b9a82a4ec88f68f23b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9e26c8ffca249b9a82a4ec88f68f23b, State: Initialized, Role: FOLLOWER
I20260928 14:56:20.688675 1257 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9e26c8ffca249b9a82a4ec88f68f23b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9e26c8ffca249b9a82a4ec88f68f23b" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 43811 } }
I20260928 14:56:20.688935 1257 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9e26c8ffca249b9a82a4ec88f68f23b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:56:20.689324 1257 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9e26c8ffca249b9a82a4ec88f68f23b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:56:20.689579 1257 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9e26c8ffca249b9a82a4ec88f68f23b [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:56:20.692047 1257 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9e26c8ffca249b9a82a4ec88f68f23b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9e26c8ffca249b9a82a4ec88f68f23b" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 43811 } }
I20260928 14:56:20.692688 1257 leader_election.cc:302] T 00000000000000000000000000000000 P b9e26c8ffca249b9a82a4ec88f68f23b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9e26c8ffca249b9a82a4ec88f68f23b; no voters:
I20260928 14:56:20.693324 1257 leader_election.cc:288] T 00000000000000000000000000000000 P b9e26c8ffca249b9a82a4ec88f68f23b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:56:20.693558 1262 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9e26c8ffca249b9a82a4ec88f68f23b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:56:20.695108 1262 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9e26c8ffca249b9a82a4ec88f68f23b [term 1 LEADER]: Becoming Leader. State: Replica: b9e26c8ffca249b9a82a4ec88f68f23b, State: Running, Role: LEADER
I20260928 14:56:20.695719 1262 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9e26c8ffca249b9a82a4ec88f68f23b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9e26c8ffca249b9a82a4ec88f68f23b" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 43811 } }
I20260928 14:56:20.695969 1257 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9e26c8ffca249b9a82a4ec88f68f23b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:56:20.698813 1264 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9e26c8ffca249b9a82a4ec88f68f23b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9e26c8ffca249b9a82a4ec88f68f23b. Latest consensus state: current_term: 1 leader_uuid: "b9e26c8ffca249b9a82a4ec88f68f23b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9e26c8ffca249b9a82a4ec88f68f23b" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 43811 } } }
I20260928 14:56:20.699239 1264 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9e26c8ffca249b9a82a4ec88f68f23b [sys.catalog]: This master's current role is: LEADER
I20260928 14:56:20.699820 1263 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9e26c8ffca249b9a82a4ec88f68f23b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9e26c8ffca249b9a82a4ec88f68f23b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9e26c8ffca249b9a82a4ec88f68f23b" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 43811 } } }
I20260928 14:56:20.700217 1263 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9e26c8ffca249b9a82a4ec88f68f23b [sys.catalog]: This master's current role is: LEADER
I20260928 14:56:20.703831 1270 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:56:20.711495 1270 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:56:20.722203 1270 catalog_manager.cc:1347] Generated new cluster ID: ac2bc00856ba483398fc25c05d48ef89
I20260928 14:56:20.722306 1270 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:56:20.731145 1270 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:56:20.732611 1270 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:56:20.743561 1270 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9e26c8ffca249b9a82a4ec88f68f23b: Generated new TSK 0
I20260928 14:56:20.744556 1270 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.40.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab.
Entry for principal kudu/127.31.40.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab.
I20260928 14:56:20.807696 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.1:0
--local_ip_for_outbound_sockets=127.31.40.1
--webserver_interface=127.31.40.1
--webserver_port=0
--tserver_master_addrs=127.31.40.62:43811
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab
--principal=kudu/127.31.40.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:38353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:56:21.155519 1284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:56:21.155864 1284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:56:21.156067 1284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:56:21.163314 1284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:56:21.163424 1284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:56:21.163530 1284 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 14:56:21.163586 1284 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:56:21.163656 1284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.1
I20260928 14:56:21.173544 1284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:38353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab
--principal=kudu/127.31.40.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.40.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.40.62:43811
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.1
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:56:21.174773 1284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:56:21.176138 1284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:56:21.189790 1290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:21.189764 1292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:21.189837 1289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:21.191066 1284 server_base.cc:1034] running on GCE node
I20260928 14:56:21.191648 1284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:56:21.192723 1284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:56:21.193984 1284 hybrid_clock.cc:648] HybridClock initialized: now 1790607381193922 us; error 97 us; skew 500 ppm
I20260928 14:56:21.194335 1284 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:56:21 dist-test-slave-j968 krb5kdc[1165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607381, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:56:21.197818 1284 init.cc:377] Logged in from keytab as kudu/127.31.40.1@KRBTEST.COM (short username kudu)
I20260928 14:56:21.199468 1284 webserver.cc:460] Webserver started at http://127.31.40.1:38523/ using document root <none> and password file <none>
I20260928 14:56:21.200291 1284 fs_manager.cc:359] Metadata directory not provided
I20260928 14:56:21.200433 1284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:56:21.200744 1284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:56:21.204099 1284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-0/data/instance:
uuid: "c9fce3001c8a4244b058751cace83dd0"
format_stamp: "Formatted at 2026-09-28 14:56:21 on dist-test-slave-j968"
server_key: "d1064f4e568fc13bb8d9ba1a94cd7406"
server_key_iv: "df9da92afea0ac5856f0e5679fc066f6"
server_key_version: "encryptionkey@0"
I20260928 14:56:21.204850 1284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-0/wal/instance:
uuid: "c9fce3001c8a4244b058751cace83dd0"
format_stamp: "Formatted at 2026-09-28 14:56:21 on dist-test-slave-j968"
server_key: "d1064f4e568fc13bb8d9ba1a94cd7406"
server_key_iv: "df9da92afea0ac5856f0e5679fc066f6"
server_key_version: "encryptionkey@0"
I20260928 14:56:21.210520 1284 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 14:56:21.214205 1299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:21.215736 1284 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:56:21.215953 1284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-0/wal
uuid: "c9fce3001c8a4244b058751cace83dd0"
format_stamp: "Formatted at 2026-09-28 14:56:21 on dist-test-slave-j968"
server_key: "d1064f4e568fc13bb8d9ba1a94cd7406"
server_key_iv: "df9da92afea0ac5856f0e5679fc066f6"
server_key_version: "encryptionkey@0"
I20260928 14:56:21.216193 1284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:56:21.231288 1284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:56:21.236290 1284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:56:21.236613 1284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:56:21.239552 1284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:56:21.239686 1284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:21.239820 1284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:56:21.239984 1284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Sep 28 14:56:21 dist-test-slave-j968 krb5kdc[1165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607381, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.1@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:56:21.268659 1309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:21.240984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43811 (local address 127.31.40.1:33341)
0928 14:56:21.242125 (+ 1141us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:21.242182 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:21.245016 (+ 2834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:21.245927 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:21.245944 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:21.246949 (+ 1005us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:56:21.248109 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:21.248141 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:21.249821 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:21.249834 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:21.252638 (+ 2804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:21.252663 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:21.253874 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:21.253884 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:21.254259 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:21.255159 (+ 900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:56:21.255202 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:56:21.259345 (+ 4143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:56:21.263459 (+ 4114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:21.263469 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:21.263490 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:21.264135 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:21.264511 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:21.264517 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:21.264520 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:21.264748 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:21.265312 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:56:21.265320 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:56:21.265898 (+ 578us) client_negotiation.cc:769] Sending connection context
0928 14:56:21.266224 (+ 326us) client_negotiation.cc:240] Negotiation successful
0928 14:56:21.266911 (+ 687us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":823,"thread_start_us":557,"threads_started":1}
I20260928 14:56:21.269876 1308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:21.240131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:33341 (local address 127.31.40.62:43811)
0928 14:56:21.241559 (+ 1428us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:21.241569 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:21.242335 (+ 766us) server_negotiation.cc:407] Connection header received
0928 14:56:21.245396 (+ 3061us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:21.245402 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:21.245485 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:21.245865 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:21.248341 (+ 2476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:21.249619 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:21.252820 (+ 3201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:21.253592 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:21.253707 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:21.259752 (+ 6045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:21.259757 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:21.259760 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:21.259795 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:21.263276 (+ 3481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:21.264264 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:21.264270 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:21.264274 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:21.264401 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:21.264893 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:21.264898 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:21.264901 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:21.265184 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:21.265340 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:21.269336 (+ 3996us) server_negotiation.cc:299] Negotiation successful
0928 14:56:21.269550 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1254,"thread_start_us":1136,"threads_started":1}
I20260928 14:56:21.282316 1284 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.1:36267
I20260928 14:56:21.282364 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.1:36267 every 8 connection(s)
I20260928 14:56:21.283998 1284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
I20260928 14:56:21.286484 1421 heartbeater.cc:347] Connected to a master server at 127.31.40.62:43811
I20260928 14:56:21.286773 1421 heartbeater.cc:464] Registering TS with master...
I20260928 14:56:21.287554 1421 heartbeater.cc:511] Master 127.31.40.62:43811 requested a full tablet report, sending...
I20260928 14:56:21.290045 1211 ts_manager.cc:194] Registered new tserver with Master: c9fce3001c8a4244b058751cace83dd0 (127.31.40.1:36267)
I20260928 14:56:21.290709 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 1284
I20260928 14:56:21.290916 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-0/wal/instance
I20260928 14:56:21.291702 31904 external_mini_cluster.cc:1428] Setting key fb2c65647ca5eb1192f39030bee75e2c
I20260928 14:56:21.292364 1410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:21.276843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43811 (local address 127.31.40.1:42427)
0928 14:56:21.277625 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:21.277642 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:21.277814 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:21.278188 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:21.278192 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:21.278853 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:56:21.279378 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:21.279398 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:21.280969 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:21.280977 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:21.282679 (+ 1702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:21.282693 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:21.283727 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:21.283732 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:21.283993 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:21.284764 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:56:21.284796 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:56:21.286086 (+ 1290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:56:21.289351 (+ 3265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:21.289356 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:21.289359 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:21.289878 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:21.290349 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:21.290353 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:21.290355 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:21.290499 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:21.291207 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:56:21.291211 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:56:21.291355 (+ 144us) client_negotiation.cc:769] Sending connection context
0928 14:56:21.291493 (+ 138us) client_negotiation.cc:240] Negotiation successful
0928 14:56:21.291815 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":751,"threads_started":1}
I20260928 14:56:21.292389 1308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:21.277354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:42427 (local address 127.31.40.62:43811)
0928 14:56:21.277590 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:21.277598 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:21.277742 (+ 144us) server_negotiation.cc:407] Connection header received
0928 14:56:21.277968 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:21.277973 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:21.278046 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:21.278146 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:21.279587 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:21.280783 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:21.282821 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:21.283491 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:21.283641 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:21.286322 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:21.286328 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:21.286333 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:21.286371 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:21.289220 (+ 2849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:21.289989 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:21.289993 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:21.289995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:21.290214 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:21.290613 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:21.290619 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:21.290623 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:21.290995 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:21.291153 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:21.291639 (+ 486us) server_negotiation.cc:299] Negotiation successful
0928 14:56:21.291940 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 14:56:21.292294 1211 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.1@KRBTEST.COM'} at 127.31.40.1:33341
WARNING: no policy specified for kudu/127.31.40.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab.
Entry for principal kudu/127.31.40.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab.
I20260928 14:56:21.334923 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.2:0
--local_ip_for_outbound_sockets=127.31.40.2
--webserver_interface=127.31.40.2
--webserver_port=0
--tserver_master_addrs=127.31.40.62:43811
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab
--principal=kudu/127.31.40.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:38353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:56:21.671546 1425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:56:21.671924 1425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:56:21.672142 1425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:56:21.681977 1425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:56:21.682152 1425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:56:21.682325 1425 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 14:56:21.682415 1425 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:56:21.682524 1425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.2
I20260928 14:56:21.693470 1425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:38353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab
--principal=kudu/127.31.40.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.40.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.40.62:43811
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.2
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:56:21.694752 1425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:56:21.696188 1425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:56:21.709802 1430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:21.709811 1431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:21.709807 1433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:21.710729 1425 server_base.cc:1034] running on GCE node
I20260928 14:56:21.711377 1425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:56:21.712824 1425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:56:21.714118 1425 hybrid_clock.cc:648] HybridClock initialized: now 1790607381714042 us; error 68 us; skew 500 ppm
I20260928 14:56:21.714496 1425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:56:21 dist-test-slave-j968 krb5kdc[1165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607381, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:56:21.718237 1425 init.cc:377] Logged in from keytab as kudu/127.31.40.2@KRBTEST.COM (short username kudu)
I20260928 14:56:21.719936 1425 webserver.cc:460] Webserver started at http://127.31.40.2:46433/ using document root <none> and password file <none>
I20260928 14:56:21.720690 1425 fs_manager.cc:359] Metadata directory not provided
I20260928 14:56:21.720839 1425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:56:21.721202 1425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:56:21.724551 1425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-1/data/instance:
uuid: "85d458ba5c774fca927a3fcd6362fb71"
format_stamp: "Formatted at 2026-09-28 14:56:21 on dist-test-slave-j968"
server_key: "040d35aa81949a27c6fd277f699e3790"
server_key_iv: "591837e7d0bca7e95007765de53de203"
server_key_version: "encryptionkey@0"
I20260928 14:56:21.725454 1425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-1/wal/instance:
uuid: "85d458ba5c774fca927a3fcd6362fb71"
format_stamp: "Formatted at 2026-09-28 14:56:21 on dist-test-slave-j968"
server_key: "040d35aa81949a27c6fd277f699e3790"
server_key_iv: "591837e7d0bca7e95007765de53de203"
server_key_version: "encryptionkey@0"
I20260928 14:56:21.731666 1425 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260928 14:56:21.736797 1440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:21.738958 1425 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 14:56:21.739204 1425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-1/wal
uuid: "85d458ba5c774fca927a3fcd6362fb71"
format_stamp: "Formatted at 2026-09-28 14:56:21 on dist-test-slave-j968"
server_key: "040d35aa81949a27c6fd277f699e3790"
server_key_iv: "591837e7d0bca7e95007765de53de203"
server_key_version: "encryptionkey@0"
I20260928 14:56:21.739460 1425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:56:21.769490 1425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:56:21.774688 1425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:56:21.775017 1425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:56:21.778046 1425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:56:21.778198 1425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:21.778327 1425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:56:21.778419 1425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 14:56:21 dist-test-slave-j968 krb5kdc[1165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607381, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.2@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:56:21.804690 1308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:21.778672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:45367 (local address 127.31.40.62:43811)
0928 14:56:21.778883 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:21.778889 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:21.780743 (+ 1854us) server_negotiation.cc:407] Connection header received
0928 14:56:21.782936 (+ 2193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:21.782941 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:21.783042 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:21.783169 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:21.785725 (+ 2556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:21.786520 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:21.788691 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:21.789512 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:21.789675 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:21.795280 (+ 5605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:21.795287 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:21.795292 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:21.795343 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:21.799653 (+ 4310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:21.800943 (+ 1290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:21.800949 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:21.800953 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:21.801100 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:21.801577 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:21.801583 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:21.801586 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:21.802036 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:21.802201 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:21.804097 (+ 1896us) server_negotiation.cc:299] Negotiation successful
0928 14:56:21.804335 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 14:56:21.805435 1449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:21.779347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43811 (local address 127.31.40.2:45367)
0928 14:56:21.780519 (+ 1172us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:21.780571 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:21.782589 (+ 2018us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:21.783481 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:21.783497 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:21.784369 (+ 872us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:56:21.785468 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:21.785489 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:21.786729 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:21.786737 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:21.788477 (+ 1740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:21.788501 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:21.790064 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:21.790072 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:21.790507 (+ 435us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:21.791412 (+ 905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:56:21.791472 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:56:21.794981 (+ 3509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:56:21.799843 (+ 4862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:21.799851 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:21.799915 (+ 64us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:21.800794 (+ 879us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:21.801222 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:21.801227 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:21.801230 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:21.801448 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:21.802171 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:56:21.802179 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:56:21.802601 (+ 422us) client_negotiation.cc:769] Sending connection context
0928 14:56:21.803003 (+ 402us) client_negotiation.cc:240] Negotiation successful
0928 14:56:21.803653 (+ 650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":842,"thread_start_us":599,"threads_started":1}
I20260928 14:56:21.825327 1425 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.2:39879
I20260928 14:56:21.825416 1560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.2:39879 every 8 connection(s)
I20260928 14:56:21.826704 1425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
I20260928 14:56:21.829684 1561 heartbeater.cc:347] Connected to a master server at 127.31.40.62:43811
I20260928 14:56:21.829946 1561 heartbeater.cc:464] Registering TS with master...
I20260928 14:56:21.830639 1561 heartbeater.cc:511] Master 127.31.40.62:43811 requested a full tablet report, sending...
I20260928 14:56:21.832116 1210 ts_manager.cc:194] Registered new tserver with Master: 85d458ba5c774fca927a3fcd6362fb71 (127.31.40.2:39879)
I20260928 14:56:21.833261 1210 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.2@KRBTEST.COM'} at 127.31.40.2:45367
I20260928 14:56:21.833563 1545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:21.818958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43811 (local address 127.31.40.2:45297)
0928 14:56:21.819785 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:21.819804 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:21.820034 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:21.820395 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:21.820399 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:21.821178 (+ 779us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:56:21.821778 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:21.821798 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:21.823235 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:21.823241 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:21.824588 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:21.824604 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:21.825647 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:21.825653 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:21.825779 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:21.826411 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:56:21.826429 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:56:21.827286 (+ 857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:56:21.830875 (+ 3589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:21.830881 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:21.830885 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:21.831285 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:21.831838 (+ 553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:21.831841 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:21.831844 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:21.832023 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:21.832706 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:56:21.832709 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:56:21.832860 (+ 151us) client_negotiation.cc:769] Sending connection context
0928 14:56:21.832967 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 14:56:21.833195 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"thread_start_us":509,"threads_started":1}
I20260928 14:56:21.833652 1308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:21.818867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:45297 (local address 127.31.40.62:43811)
0928 14:56:21.819099 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:21.819107 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:21.819926 (+ 819us) server_negotiation.cc:407] Connection header received
0928 14:56:21.820192 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:21.820196 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:21.820261 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:21.820360 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:21.822013 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:21.823090 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:21.824731 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:21.825223 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:21.825363 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:21.827471 (+ 2108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:21.827477 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:21.827482 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:21.827524 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:21.830731 (+ 3207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:21.831549 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:21.831553 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:21.831559 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:21.831724 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:21.832139 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:21.832145 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:21.832148 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:21.832592 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:21.832737 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:21.833109 (+ 372us) server_negotiation.cc:299] Negotiation successful
0928 14:56:21.833340 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 14:56:21.834685 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 1425
I20260928 14:56:21.834858 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-1/wal/instance
I20260928 14:56:21.835330 31904 external_mini_cluster.cc:1428] Setting key 2e271f80abbeb00decd70d5543b41dba
WARNING: no policy specified for kudu/127.31.40.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab.
Entry for principal kudu/127.31.40.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab.
I20260928 14:56:21.877349 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.3:0
--local_ip_for_outbound_sockets=127.31.40.3
--webserver_interface=127.31.40.3
--webserver_port=0
--tserver_master_addrs=127.31.40.62:43811
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab
--principal=kudu/127.31.40.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:38353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:56:22.214704 1565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:56:22.215039 1565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:56:22.215211 1565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:56:22.222740 1565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:56:22.222930 1565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:56:22.223080 1565 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 14:56:22.223168 1565 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:56:22.223304 1565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.3
I20260928 14:56:22.233850 1565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:38353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab
--principal=kudu/127.31.40.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.40.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.40.62:43811
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.3
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:56:22.235267 1565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:56:22.236725 1565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:56:22.251072 1573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:22.251372 1571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:22.251339 1570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:22.252588 1565 server_base.cc:1034] running on GCE node
I20260928 14:56:22.253257 1565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:56:22.254298 1565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:56:22.255563 1565 hybrid_clock.cc:648] HybridClock initialized: now 1790607382255493 us; error 58 us; skew 500 ppm
I20260928 14:56:22.255977 1565 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:56:22 dist-test-slave-j968 krb5kdc[1165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607382, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:56:22.259333 1565 init.cc:377] Logged in from keytab as kudu/127.31.40.3@KRBTEST.COM (short username kudu)
I20260928 14:56:22.260921 1565 webserver.cc:460] Webserver started at http://127.31.40.3:36805/ using document root <none> and password file <none>
I20260928 14:56:22.261627 1565 fs_manager.cc:359] Metadata directory not provided
I20260928 14:56:22.261745 1565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:56:22.262089 1565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:56:22.266227 1565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-2/data/instance:
uuid: "a3466cf67681447890434b1fc93bba03"
format_stamp: "Formatted at 2026-09-28 14:56:22 on dist-test-slave-j968"
server_key: "c45a1c46ddab5685ffb91329eeb18f92"
server_key_iv: "fe74bd33c9912a5aaca956e4edc1c942"
server_key_version: "encryptionkey@0"
I20260928 14:56:22.267153 1565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-2/wal/instance:
uuid: "a3466cf67681447890434b1fc93bba03"
format_stamp: "Formatted at 2026-09-28 14:56:22 on dist-test-slave-j968"
server_key: "c45a1c46ddab5685ffb91329eeb18f92"
server_key_iv: "fe74bd33c9912a5aaca956e4edc1c942"
server_key_version: "encryptionkey@0"
I20260928 14:56:22.273233 1565 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260928 14:56:22.277244 1580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:22.278757 1565 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 14:56:22.279003 1565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-2/wal
uuid: "a3466cf67681447890434b1fc93bba03"
format_stamp: "Formatted at 2026-09-28 14:56:22 on dist-test-slave-j968"
server_key: "c45a1c46ddab5685ffb91329eeb18f92"
server_key_iv: "fe74bd33c9912a5aaca956e4edc1c942"
server_key_version: "encryptionkey@0"
I20260928 14:56:22.279214 1565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:56:22.296540 1421 heartbeater.cc:503] Master 127.31.40.62:43811 was elected leader, sending a full tablet report...
I20260928 14:56:22.306190 1565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:56:22.311193 1565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:56:22.311465 1565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:56:22.314455 1565 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:56:22.314618 1565 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:22.314754 1565 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:56:22.314844 1565 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 14:56:22 dist-test-slave-j968 krb5kdc[1165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607382, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.3@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:56:22.341876 1308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:22.315368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:54325 (local address 127.31.40.62:43811)
0928 14:56:22.315573 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:22.315579 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:22.317511 (+ 1932us) server_negotiation.cc:407] Connection header received
0928 14:56:22.319968 (+ 2457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:22.319974 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:22.320077 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:22.320213 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:22.322957 (+ 2744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:22.324196 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:22.327393 (+ 3197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:22.328159 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:22.328341 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:22.334036 (+ 5695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:22.334042 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:22.334046 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:22.334085 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:22.337630 (+ 3545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:22.338591 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:22.338595 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:22.338598 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:22.338728 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:22.339208 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:22.339214 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:22.339218 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:22.339545 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:22.339729 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:22.340870 (+ 1141us) server_negotiation.cc:299] Negotiation successful
0928 14:56:22.341251 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 14:56:22.342725 1589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:22.316299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43811 (local address 127.31.40.3:54325)
0928 14:56:22.317297 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:22.317347 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:22.319592 (+ 2245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:22.320571 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:22.320588 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:22.321599 (+ 1011us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:56:22.322674 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:22.322713 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:22.324430 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:22.324441 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:22.327203 (+ 2762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:22.327226 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:22.328518 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:22.328532 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:22.329043 (+ 511us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:22.330135 (+ 1092us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:56:22.330172 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:56:22.333700 (+ 3528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:56:22.337856 (+ 4156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:22.337867 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:22.337883 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:22.338386 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:22.338864 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:22.338869 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:22.338873 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:22.339080 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:22.339696 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:56:22.339707 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:56:22.340269 (+ 562us) client_negotiation.cc:769] Sending connection context
0928 14:56:22.340664 (+ 395us) client_negotiation.cc:240] Negotiation successful
0928 14:56:22.341218 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"thread_start_us":427,"threads_started":1}
I20260928 14:56:22.367062 1308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:22.350632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:56499 (local address 127.31.40.62:43811)
0928 14:56:22.350823 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:22.350828 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:22.351581 (+ 753us) server_negotiation.cc:407] Connection header received
0928 14:56:22.351815 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:22.351820 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:22.351910 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:22.352028 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:22.353680 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:22.354634 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:22.357778 (+ 3144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:22.358311 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:22.358476 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:22.360748 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:22.360753 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:22.360757 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:22.360798 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:22.363962 (+ 3164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:22.364988 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:22.364992 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:22.364994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:22.365113 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:22.365473 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:22.365478 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:22.365482 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:22.365772 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:22.365936 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:22.366343 (+ 407us) server_negotiation.cc:299] Negotiation successful
0928 14:56:22.366574 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 14:56:22.367090 1651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:22.350759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43811 (local address 127.31.40.3:56499)
0928 14:56:22.351449 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:22.351471 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:22.351669 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:22.352191 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:22.352197 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:22.352885 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:56:22.353517 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:22.353535 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:22.354816 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:22.354825 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:22.357568 (+ 2743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:22.357589 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:22.358723 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:22.358727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:22.358838 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:22.359480 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:56:22.359498 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:56:22.360495 (+ 997us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:56:22.364148 (+ 3653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:22.364153 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:22.364159 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:22.364825 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:22.365223 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:22.365227 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:22.365231 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:22.365366 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:22.365900 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:56:22.365905 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:56:22.366110 (+ 205us) client_negotiation.cc:769] Sending connection context
0928 14:56:22.366249 (+ 139us) client_negotiation.cc:240] Negotiation successful
0928 14:56:22.366605 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":403,"threads_started":1}
I20260928 14:56:22.378805 1565 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.3:33391
I20260928 14:56:22.378834 1700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.3:33391 every 8 connection(s)
I20260928 14:56:22.380180 1565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
I20260928 14:56:22.382503 1701 heartbeater.cc:347] Connected to a master server at 127.31.40.62:43811
I20260928 14:56:22.382754 1701 heartbeater.cc:464] Registering TS with master...
I20260928 14:56:22.383548 1701 heartbeater.cc:511] Master 127.31.40.62:43811 requested a full tablet report, sending...
I20260928 14:56:22.385130 1210 ts_manager.cc:194] Registered new tserver with Master: a3466cf67681447890434b1fc93bba03 (127.31.40.3:33391)
I20260928 14:56:22.386056 1210 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.3@KRBTEST.COM'} at 127.31.40.3:54325
I20260928 14:56:22.387492 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 1565
I20260928 14:56:22.387689 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-2/wal/instance
I20260928 14:56:22.388335 31904 external_mini_cluster.cc:1428] Setting key ee70366cf7817cafd5933903c49ba5b8
I20260928 14:56:22.391651 31904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 14:56:22 dist-test-slave-j968 krb5kdc[1165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 14:56:22 dist-test-slave-j968 krb5kdc[1165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:56:22.443145 1308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:22.427062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43832 (local address 127.31.40.62:43811)
0928 14:56:22.427348 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:22.427356 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:22.427576 (+ 220us) server_negotiation.cc:407] Connection header received
0928 14:56:22.427730 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:22.427734 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:22.427802 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:22.427947 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:22.429992 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:22.431002 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:22.432984 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:22.433453 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:22.433567 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:22.437588 (+ 4021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:22.437593 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:22.437597 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:22.437633 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:22.440389 (+ 2756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:22.441117 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:22.441121 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:22.441123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:22.441252 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:22.441621 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:22.441624 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:22.441626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:22.441912 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:22.442119 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:22.442444 (+ 325us) server_negotiation.cc:299] Negotiation successful
0928 14:56:22.442631 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260928 14:56:22.448362 1210 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43832:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:56:22.455394 1210 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:56:22.481174 1718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:22.469570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49986 (local address 127.31.40.3:33391)
0928 14:56:22.470858 (+ 1288us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:22.470871 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:22.470899 (+ 28us) server_negotiation.cc:407] Connection header received
0928 14:56:22.470998 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:22.471009 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:22.471291 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:22.471574 (+ 283us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:56:22.472942 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:22.474647 (+ 1705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:22.478447 (+ 3800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:22.479765 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:22.479916 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:22.480175 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:22.480641 (+ 466us) server_negotiation.cc:299] Negotiation successful
0928 14:56:22.480808 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1109,"thread_start_us":1014,"threads_started":1}
I20260928 14:56:22.481268 1716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:22.469085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.3:33391 (local address 127.0.0.1:49986)
0928 14:56:22.469970 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:22.470013 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:22.470176 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:22.471967 (+ 1791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:22.471976 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:22.472011 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:56:22.472597 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:22.472634 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:22.474771 (+ 2137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:22.474777 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:22.478249 (+ 3472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:22.478264 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:22.480272 (+ 2008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:22.480277 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:22.480405 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:22.480446 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 14:56:22.480604 (+ 158us) client_negotiation.cc:240] Negotiation successful
0928 14:56:22.480809 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"mutex_wait_us":120,"thread_start_us":3530,"threads_started":1}
I20260928 14:56:22.481616 1714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:22.468773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48438 (local address 127.31.40.1:36267)
0928 14:56:22.469367 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:22.469376 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:22.470794 (+ 1418us) server_negotiation.cc:407] Connection header received
0928 14:56:22.471138 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:22.471145 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:22.471316 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:22.471530 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:56:22.473411 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:22.474722 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:22.478451 (+ 3729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:22.480032 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:22.480151 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:22.480343 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:22.481133 (+ 790us) server_negotiation.cc:299] Negotiation successful
0928 14:56:22.481320 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":480,"threads_started":1}
I20260928 14:56:22.481904 1715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:22.468847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43434 (local address 127.31.40.2:39879)
0928 14:56:22.472492 (+ 3645us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:22.472501 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:22.473249 (+ 748us) server_negotiation.cc:407] Connection header received
0928 14:56:22.473392 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:22.473400 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:22.473670 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:22.473884 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:56:22.474454 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:22.475913 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:22.478366 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:22.479971 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:22.480123 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:22.480344 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:22.481357 (+ 1013us) server_negotiation.cc:299] Negotiation successful
0928 14:56:22.481545 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3459,"thread_start_us":3321,"threads_started":1}
I20260928 14:56:22.482581 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:22.469939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.2:39879 (local address 127.0.0.1:43434)
0928 14:56:22.472964 (+ 3025us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:22.473001 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:22.473134 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:22.473881 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:22.473886 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:22.473922 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:56:22.474270 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:22.474281 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:22.476205 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:22.476211 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:22.478195 (+ 1984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:22.478212 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:22.480575 (+ 2363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:22.480581 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:22.480820 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:22.480944 (+ 124us) client_negotiation.cc:769] Sending connection context
0928 14:56:22.481043 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 14:56:22.481189 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2860,"mutex_wait_us":172,"thread_start_us":419,"threads_started":1}
I20260928 14:56:22.482803 1719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:22.469118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:36267 (local address 127.0.0.1:48438)
0928 14:56:22.470668 (+ 1550us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:22.470699 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:22.470959 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:22.471992 (+ 1033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:22.471997 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:22.472035 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:56:22.473165 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:22.473179 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:22.474847 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:22.474854 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:22.478249 (+ 3395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:22.478264 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:22.480546 (+ 2282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:22.480556 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:22.480813 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:22.480938 (+ 125us) client_negotiation.cc:769] Sending connection context
0928 14:56:22.482173 (+ 1235us) client_negotiation.cc:240] Negotiation successful
0928 14:56:22.482342 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1421,"mutex_wait_us":758,"spinlock_wait_cycles":2304,"thread_start_us":473,"threads_started":1}
I20260928 14:56:22.485684 1631 tablet_service.cc:1467] Processing CreateTablet for tablet 6732be1dc9f94e398737050018dfda7b (DEFAULT_TABLE table=test-table [id=d7d08e383fc94168bd9a40a5bc5ceb30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:56:22.486068 1351 tablet_service.cc:1467] Processing CreateTablet for tablet 6732be1dc9f94e398737050018dfda7b (DEFAULT_TABLE table=test-table [id=d7d08e383fc94168bd9a40a5bc5ceb30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:56:22.486229 1491 tablet_service.cc:1467] Processing CreateTablet for tablet 6732be1dc9f94e398737050018dfda7b (DEFAULT_TABLE table=test-table [id=d7d08e383fc94168bd9a40a5bc5ceb30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:56:22.487313 1351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6732be1dc9f94e398737050018dfda7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:56:22.487327 1631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6732be1dc9f94e398737050018dfda7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:56:22.487495 1491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6732be1dc9f94e398737050018dfda7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:56:22.500545 1720 tablet_bootstrap.cc:492] T 6732be1dc9f94e398737050018dfda7b P c9fce3001c8a4244b058751cace83dd0: Bootstrap starting.
I20260928 14:56:22.504627 1722 tablet_bootstrap.cc:492] T 6732be1dc9f94e398737050018dfda7b P 85d458ba5c774fca927a3fcd6362fb71: Bootstrap starting.
I20260928 14:56:22.505381 1720 tablet_bootstrap.cc:654] T 6732be1dc9f94e398737050018dfda7b P c9fce3001c8a4244b058751cace83dd0: Neither blocks nor log segments found. Creating new log.
I20260928 14:56:22.505986 1721 tablet_bootstrap.cc:492] T 6732be1dc9f94e398737050018dfda7b P a3466cf67681447890434b1fc93bba03: Bootstrap starting.
I20260928 14:56:22.506969 1720 log.cc:824] T 6732be1dc9f94e398737050018dfda7b P c9fce3001c8a4244b058751cace83dd0: Log is configured to *not* fsync() on all Append() calls
I20260928 14:56:22.509896 1721 tablet_bootstrap.cc:654] T 6732be1dc9f94e398737050018dfda7b P a3466cf67681447890434b1fc93bba03: Neither blocks nor log segments found. Creating new log.
I20260928 14:56:22.510241 1720 tablet_bootstrap.cc:492] T 6732be1dc9f94e398737050018dfda7b P c9fce3001c8a4244b058751cace83dd0: No bootstrap required, opened a new log
I20260928 14:56:22.510526 1720 ts_tablet_manager.cc:1397] T 6732be1dc9f94e398737050018dfda7b P c9fce3001c8a4244b058751cace83dd0: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260928 14:56:22.511081 1722 tablet_bootstrap.cc:654] T 6732be1dc9f94e398737050018dfda7b P 85d458ba5c774fca927a3fcd6362fb71: Neither blocks nor log segments found. Creating new log.
I20260928 14:56:22.511188 1721 log.cc:824] T 6732be1dc9f94e398737050018dfda7b P a3466cf67681447890434b1fc93bba03: Log is configured to *not* fsync() on all Append() calls
I20260928 14:56:22.512578 1722 log.cc:824] T 6732be1dc9f94e398737050018dfda7b P 85d458ba5c774fca927a3fcd6362fb71: Log is configured to *not* fsync() on all Append() calls
I20260928 14:56:22.515527 1722 tablet_bootstrap.cc:492] T 6732be1dc9f94e398737050018dfda7b P 85d458ba5c774fca927a3fcd6362fb71: No bootstrap required, opened a new log
I20260928 14:56:22.515820 1722 ts_tablet_manager.cc:1397] T 6732be1dc9f94e398737050018dfda7b P 85d458ba5c774fca927a3fcd6362fb71: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20260928 14:56:22.516069 1721 tablet_bootstrap.cc:492] T 6732be1dc9f94e398737050018dfda7b P a3466cf67681447890434b1fc93bba03: No bootstrap required, opened a new log
I20260928 14:56:22.516341 1721 ts_tablet_manager.cc:1397] T 6732be1dc9f94e398737050018dfda7b P a3466cf67681447890434b1fc93bba03: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.008s
I20260928 14:56:22.521387 1720 raft_consensus.cc:348] T 6732be1dc9f94e398737050018dfda7b P c9fce3001c8a4244b058751cace83dd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 } } peers { permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 } } peers { permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 } }
I20260928 14:56:22.521914 1720 raft_consensus.cc:374] T 6732be1dc9f94e398737050018dfda7b P c9fce3001c8a4244b058751cace83dd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:56:22.522061 1720 raft_consensus.cc:729] T 6732be1dc9f94e398737050018dfda7b P c9fce3001c8a4244b058751cace83dd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9fce3001c8a4244b058751cace83dd0, State: Initialized, Role: FOLLOWER
I20260928 14:56:22.522846 1720 consensus_queue.cc:260] T 6732be1dc9f94e398737050018dfda7b P c9fce3001c8a4244b058751cace83dd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 } } peers { permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 } } peers { permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 } }
I20260928 14:56:22.523918 1722 raft_consensus.cc:348] T 6732be1dc9f94e398737050018dfda7b P 85d458ba5c774fca927a3fcd6362fb71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 } } peers { permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 } } peers { permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 } }
I20260928 14:56:22.524362 1722 raft_consensus.cc:374] T 6732be1dc9f94e398737050018dfda7b P 85d458ba5c774fca927a3fcd6362fb71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:56:22.524477 1722 raft_consensus.cc:729] T 6732be1dc9f94e398737050018dfda7b P 85d458ba5c774fca927a3fcd6362fb71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85d458ba5c774fca927a3fcd6362fb71, State: Initialized, Role: FOLLOWER
I20260928 14:56:22.525198 1720 ts_tablet_manager.cc:1428] T 6732be1dc9f94e398737050018dfda7b P c9fce3001c8a4244b058751cace83dd0: Time spent starting tablet: real 0.015s user 0.010s sys 0.005s
I20260928 14:56:22.525071 1722 consensus_queue.cc:260] T 6732be1dc9f94e398737050018dfda7b P 85d458ba5c774fca927a3fcd6362fb71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 } } peers { permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 } } peers { permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 } }
I20260928 14:56:22.526314 1561 heartbeater.cc:503] Master 127.31.40.62:43811 was elected leader, sending a full tablet report...
I20260928 14:56:22.526801 1722 ts_tablet_manager.cc:1428] T 6732be1dc9f94e398737050018dfda7b P 85d458ba5c774fca927a3fcd6362fb71: Time spent starting tablet: real 0.011s user 0.008s sys 0.002s
I20260928 14:56:22.527146 1721 raft_consensus.cc:348] T 6732be1dc9f94e398737050018dfda7b P a3466cf67681447890434b1fc93bba03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 } } peers { permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 } } peers { permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 } }
I20260928 14:56:22.527581 1721 raft_consensus.cc:374] T 6732be1dc9f94e398737050018dfda7b P a3466cf67681447890434b1fc93bba03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:56:22.527709 1721 raft_consensus.cc:729] T 6732be1dc9f94e398737050018dfda7b P a3466cf67681447890434b1fc93bba03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3466cf67681447890434b1fc93bba03, State: Initialized, Role: FOLLOWER
I20260928 14:56:22.528399 1721 consensus_queue.cc:260] T 6732be1dc9f94e398737050018dfda7b P a3466cf67681447890434b1fc93bba03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 } } peers { permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 } } peers { permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 } }
I20260928 14:56:22.529723 1701 heartbeater.cc:503] Master 127.31.40.62:43811 was elected leader, sending a full tablet report...
I20260928 14:56:22.530378 1721 ts_tablet_manager.cc:1428] T 6732be1dc9f94e398737050018dfda7b P a3466cf67681447890434b1fc93bba03: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
W20260928 14:56:22.536595 1422 tablet.cc:2369] T 6732be1dc9f94e398737050018dfda7b P c9fce3001c8a4244b058751cace83dd0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:56:22.561841 1727 raft_consensus.cc:482] T 6732be1dc9f94e398737050018dfda7b P 85d458ba5c774fca927a3fcd6362fb71 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:56:22.562390 1727 raft_consensus.cc:504] T 6732be1dc9f94e398737050018dfda7b P 85d458ba5c774fca927a3fcd6362fb71 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 } } peers { permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 } } peers { permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 } }
I20260928 14:56:22.564375 1727 leader_election.cc:288] T 6732be1dc9f94e398737050018dfda7b P 85d458ba5c774fca927a3fcd6362fb71 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9fce3001c8a4244b058751cace83dd0 (127.31.40.1:36267), a3466cf67681447890434b1fc93bba03 (127.31.40.3:33391)
I20260928 14:56:22.566910 1728 raft_consensus.cc:482] T 6732be1dc9f94e398737050018dfda7b P a3466cf67681447890434b1fc93bba03 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:56:22.567224 1728 raft_consensus.cc:504] T 6732be1dc9f94e398737050018dfda7b P a3466cf67681447890434b1fc93bba03 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 } } peers { permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 } } peers { permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 } }
I20260928 14:56:22.569411 1728 leader_election.cc:288] T 6732be1dc9f94e398737050018dfda7b P a3466cf67681447890434b1fc93bba03 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9fce3001c8a4244b058751cace83dd0 (127.31.40.1:36267), 85d458ba5c774fca927a3fcd6362fb71 (127.31.40.2:39879)
I20260928 14:56:22.574568 1714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:22.565005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:41669 (local address 127.31.40.1:36267)
0928 14:56:22.565266 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:22.565273 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:22.566036 (+ 763us) server_negotiation.cc:407] Connection header received
0928 14:56:22.566293 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:22.566299 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:22.566392 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:22.566519 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:56:22.567305 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:22.568401 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:22.571486 (+ 3085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:22.573112 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:22.573245 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:22.573369 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:22.574115 (+ 746us) server_negotiation.cc:299] Negotiation successful
0928 14:56:22.574277 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260928 14:56:22.574800 1718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:22.565019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:47135 (local address 127.31.40.3:33391)
0928 14:56:22.565266 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:22.565273 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:22.566536 (+ 1263us) server_negotiation.cc:407] Connection header received
0928 14:56:22.566842 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:22.566903 (+ 61us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:22.567006 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:22.567114 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:56:22.567857 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:22.568878 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:22.572118 (+ 3240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:22.573538 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:22.573648 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:22.573763 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:22.574347 (+ 584us) server_negotiation.cc:299] Negotiation successful
0928 14:56:22.574515 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 14:56:22.575721 1730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:22.565122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.3:33391 (local address 127.31.40.2:47135)
0928 14:56:22.566428 (+ 1306us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:22.566447 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:22.566661 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:22.567313 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:22.567324 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:22.567345 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:56:22.567694 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:22.567705 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:22.569038 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:22.569044 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:22.571290 (+ 2246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:22.571304 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:22.573961 (+ 2657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:22.573968 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:22.574163 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:22.574198 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 14:56:22.575068 (+ 870us) client_negotiation.cc:240] Negotiation successful
0928 14:56:22.575237 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1197,"mutex_wait_us":3,"spinlock_wait_cycles":29440,"thread_start_us":1031,"threads_started":1}
I20260928 14:56:22.577144 1729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:22.565095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:36267 (local address 127.31.40.2:41669)
0928 14:56:22.565914 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:22.565932 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:22.566128 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:22.566537 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:22.566542 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:22.566570 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:56:22.567087 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:22.567104 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:22.568548 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:22.568555 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:22.571148 (+ 2593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:22.571221 (+ 73us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:22.573641 (+ 2420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:22.573648 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:22.573875 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:22.573926 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 14:56:22.576542 (+ 2616us) client_negotiation.cc:240] Negotiation successful
0928 14:56:22.576703 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"mutex_wait_us":82,"spinlock_wait_cycles":36736,"thread_start_us":479,"threads_started":1}
I20260928 14:56:22.577000 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6732be1dc9f94e398737050018dfda7b" candidate_uuid: "85d458ba5c774fca927a3fcd6362fb71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3466cf67681447890434b1fc93bba03" is_pre_election: true
I20260928 14:56:22.577644 1656 raft_consensus.cc:2454] T 6732be1dc9f94e398737050018dfda7b P a3466cf67681447890434b1fc93bba03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85d458ba5c774fca927a3fcd6362fb71 in term 0.
I20260928 14:56:22.578613 1442 leader_election.cc:302] T 6732be1dc9f94e398737050018dfda7b P 85d458ba5c774fca927a3fcd6362fb71 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85d458ba5c774fca927a3fcd6362fb71, a3466cf67681447890434b1fc93bba03; no voters:
I20260928 14:56:22.578487 1371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6732be1dc9f94e398737050018dfda7b" candidate_uuid: "85d458ba5c774fca927a3fcd6362fb71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9fce3001c8a4244b058751cace83dd0" is_pre_election: true
I20260928 14:56:22.579171 1727 raft_consensus.cc:2781] T 6732be1dc9f94e398737050018dfda7b P 85d458ba5c774fca927a3fcd6362fb71 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:56:22.579362 1727 raft_consensus.cc:482] T 6732be1dc9f94e398737050018dfda7b P 85d458ba5c774fca927a3fcd6362fb71 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:56:22.579298 1371 raft_consensus.cc:2454] T 6732be1dc9f94e398737050018dfda7b P c9fce3001c8a4244b058751cace83dd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85d458ba5c774fca927a3fcd6362fb71 in term 0.
I20260928 14:56:22.579475 1727 raft_consensus.cc:3037] T 6732be1dc9f94e398737050018dfda7b P 85d458ba5c774fca927a3fcd6362fb71 [term 0 FOLLOWER]: Advancing to term 1
W20260928 14:56:22.579173 1562 tablet.cc:2369] T 6732be1dc9f94e398737050018dfda7b P 85d458ba5c774fca927a3fcd6362fb71: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:56:22.584137 1727 raft_consensus.cc:504] T 6732be1dc9f94e398737050018dfda7b P 85d458ba5c774fca927a3fcd6362fb71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 } } peers { permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 } } peers { permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 } }
I20260928 14:56:22.585469 1727 leader_election.cc:288] T 6732be1dc9f94e398737050018dfda7b P 85d458ba5c774fca927a3fcd6362fb71 [CANDIDATE]: Term 1 election: Requested vote from peers c9fce3001c8a4244b058751cace83dd0 (127.31.40.1:36267), a3466cf67681447890434b1fc93bba03 (127.31.40.3:33391)
I20260928 14:56:22.585518 1371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6732be1dc9f94e398737050018dfda7b" candidate_uuid: "85d458ba5c774fca927a3fcd6362fb71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9fce3001c8a4244b058751cace83dd0"
I20260928 14:56:22.585561 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6732be1dc9f94e398737050018dfda7b" candidate_uuid: "85d458ba5c774fca927a3fcd6362fb71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3466cf67681447890434b1fc93bba03"
I20260928 14:56:22.585772 1656 raft_consensus.cc:3037] T 6732be1dc9f94e398737050018dfda7b P a3466cf67681447890434b1fc93bba03 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:56:22.585763 1371 raft_consensus.cc:3037] T 6732be1dc9f94e398737050018dfda7b P c9fce3001c8a4244b058751cace83dd0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:56:22.588945 1589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:22.569660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:36267 (local address 127.31.40.3:46617)
0928 14:56:22.570018 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:22.570037 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:22.570155 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:22.575610 (+ 5455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:22.575615 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:22.575651 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:56:22.575917 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:22.575931 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:22.581729 (+ 5798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:22.581739 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:22.584111 (+ 2372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:22.584121 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:22.588158 (+ 4037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:22.588163 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:22.588389 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:22.588428 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 14:56:22.588494 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 14:56:22.588636 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224}
I20260928 14:56:22.589293 1371 raft_consensus.cc:2454] T 6732be1dc9f94e398737050018dfda7b P c9fce3001c8a4244b058751cace83dd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85d458ba5c774fca927a3fcd6362fb71 in term 1.
I20260928 14:56:22.589329 1656 raft_consensus.cc:2454] T 6732be1dc9f94e398737050018dfda7b P a3466cf67681447890434b1fc93bba03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85d458ba5c774fca927a3fcd6362fb71 in term 1.
I20260928 14:56:22.589715 1443 leader_election.cc:302] T 6732be1dc9f94e398737050018dfda7b P 85d458ba5c774fca927a3fcd6362fb71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85d458ba5c774fca927a3fcd6362fb71, c9fce3001c8a4244b058751cace83dd0; no voters:
I20260928 14:56:22.590051 1727 raft_consensus.cc:2781] T 6732be1dc9f94e398737050018dfda7b P 85d458ba5c774fca927a3fcd6362fb71 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:56:22.590583 1715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:22.570940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:46239 (local address 127.31.40.2:39879)
0928 14:56:22.571225 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:22.571234 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:22.582987 (+ 11753us) server_negotiation.cc:407] Connection header received
0928 14:56:22.583263 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:22.583269 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:22.583341 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:22.583396 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:56:22.584250 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:22.585405 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:22.587932 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:22.589226 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:22.589345 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:22.589440 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:22.590194 (+ 754us) server_negotiation.cc:299] Negotiation successful
0928 14:56:22.590316 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 14:56:22.590607 1731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:22.570001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.2:39879 (local address 127.31.40.3:46239)
0928 14:56:22.582849 (+ 12848us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:22.582874 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:22.583112 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:22.583587 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:22.583591 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:22.583616 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:56:22.584047 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:22.584061 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:22.585704 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:22.585710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:22.587638 (+ 1928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:22.587652 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:22.589788 (+ 2136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:22.589797 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:22.589994 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:22.590032 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 14:56:22.590143 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 14:56:22.590284 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12698,"thread_start_us":12496,"threads_started":1}
I20260928 14:56:22.590646 1714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:22.570433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:46617 (local address 127.31.40.1:36267)
0928 14:56:22.575263 (+ 4830us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:22.575270 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:22.575288 (+ 18us) server_negotiation.cc:407] Connection header received
0928 14:56:22.575353 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:22.575360 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:22.575443 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:22.576297 (+ 854us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:56:22.576459 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:22.581501 (+ 5042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:22.584283 (+ 2782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:22.587981 (+ 3698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:22.589320 (+ 1339us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:22.589467 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:22.590107 (+ 640us) server_negotiation.cc:299] Negotiation successful
0928 14:56:22.590265 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4725,"thread_start_us":11773,"threads_started":1}
I20260928 14:56:22.591156 1371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6732be1dc9f94e398737050018dfda7b" candidate_uuid: "a3466cf67681447890434b1fc93bba03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9fce3001c8a4244b058751cace83dd0" is_pre_election: true
I20260928 14:56:22.591118 1727 raft_consensus.cc:686] T 6732be1dc9f94e398737050018dfda7b P 85d458ba5c774fca927a3fcd6362fb71 [term 1 LEADER]: Becoming Leader. State: Replica: 85d458ba5c774fca927a3fcd6362fb71, State: Running, Role: LEADER
I20260928 14:56:22.591812 1371 raft_consensus.cc:2379] T 6732be1dc9f94e398737050018dfda7b P c9fce3001c8a4244b058751cace83dd0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a3466cf67681447890434b1fc93bba03 in current term 1: Already voted for candidate 85d458ba5c774fca927a3fcd6362fb71 in this term.
I20260928 14:56:22.591630 1511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6732be1dc9f94e398737050018dfda7b" candidate_uuid: "a3466cf67681447890434b1fc93bba03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85d458ba5c774fca927a3fcd6362fb71" is_pre_election: true
I20260928 14:56:22.592044 1727 consensus_queue.cc:237] T 6732be1dc9f94e398737050018dfda7b P 85d458ba5c774fca927a3fcd6362fb71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 } } peers { permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 } } peers { permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 } }
I20260928 14:56:22.592957 1581 leader_election.cc:302] T 6732be1dc9f94e398737050018dfda7b P a3466cf67681447890434b1fc93bba03 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a3466cf67681447890434b1fc93bba03; no voters: 85d458ba5c774fca927a3fcd6362fb71, c9fce3001c8a4244b058751cace83dd0
I20260928 14:56:22.593386 1728 raft_consensus.cc:2726] T 6732be1dc9f94e398737050018dfda7b P a3466cf67681447890434b1fc93bba03 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 14:56:22.597625 1211 catalog_manager.cc:5495] T 6732be1dc9f94e398737050018dfda7b P 85d458ba5c774fca927a3fcd6362fb71 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85d458ba5c774fca927a3fcd6362fb71 (127.31.40.2). New cstate: current_term: 1 leader_uuid: "85d458ba5c774fca927a3fcd6362fb71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 } health_report { overall_health: UNKNOWN } } }
W20260928 14:56:22.631939 1702 tablet.cc:2369] T 6732be1dc9f94e398737050018dfda7b P a3466cf67681447890434b1fc93bba03: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 14:56:22 dist-test-slave-j968 krb5kdc[1165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607380, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.62@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:56:22.673698 1308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:22.658719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43848 (local address 127.31.40.62:43811)
0928 14:56:22.658951 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:22.658959 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:22.659696 (+ 737us) server_negotiation.cc:407] Connection header received
0928 14:56:22.659934 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:22.659939 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:22.660023 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:22.660113 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:22.661715 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:22.662580 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:22.663925 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:22.664451 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:22.664558 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:22.668154 (+ 3596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:22.668160 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:22.668163 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:22.668194 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:22.670741 (+ 2547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:22.671409 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:22.671413 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:22.671416 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:22.671694 (+ 278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:22.672172 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:22.672178 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:22.672182 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:22.672454 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:22.672656 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:22.673081 (+ 425us) server_negotiation.cc:299] Negotiation successful
0928 14:56:22.673386 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":102,"server-negotiator.queue_time_us":67}
I20260928 14:56:22.673730 1740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:22.658800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43811 (local address 127.0.0.1:43848)
0928 14:56:22.659571 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:22.659591 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:22.659759 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:22.660174 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:22.660179 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:22.661170 (+ 991us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:56:22.661537 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:22.661550 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:22.662715 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:22.662719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:22.663738 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:22.663749 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:22.664630 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:22.664634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:22.664741 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:22.665389 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:56:22.665406 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:56:22.667977 (+ 2571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:56:22.670870 (+ 2893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:22.670881 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:22.670887 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:22.671289 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:22.671800 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:22.671803 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:22.671805 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:22.671955 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:22.672594 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:56:22.672605 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:56:22.672790 (+ 185us) client_negotiation.cc:769] Sending connection context
0928 14:56:22.672959 (+ 169us) client_negotiation.cc:240] Negotiation successful
0928 14:56:22.673251 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":426,"threads_started":1}
I20260928 14:56:22.679071 1211 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:43848:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 14:56:22.680274 1211 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:56:22.687016 1351 tablet_service.cc:1467] Processing CreateTablet for tablet 19571dfe91ca4ad49671b8b000f0ad80 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5fa29f30894e4f4492f44bf144a1625d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:56:22.687093 1491 tablet_service.cc:1467] Processing CreateTablet for tablet 19571dfe91ca4ad49671b8b000f0ad80 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5fa29f30894e4f4492f44bf144a1625d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:56:22.687299 1631 tablet_service.cc:1467] Processing CreateTablet for tablet 19571dfe91ca4ad49671b8b000f0ad80 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5fa29f30894e4f4492f44bf144a1625d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:56:22.687955 1351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19571dfe91ca4ad49671b8b000f0ad80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:56:22.688050 1491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19571dfe91ca4ad49671b8b000f0ad80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:56:22.688050 1631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19571dfe91ca4ad49671b8b000f0ad80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:56:22.693849 1720 tablet_bootstrap.cc:492] T 19571dfe91ca4ad49671b8b000f0ad80 P c9fce3001c8a4244b058751cace83dd0: Bootstrap starting.
I20260928 14:56:22.695158 1721 tablet_bootstrap.cc:492] T 19571dfe91ca4ad49671b8b000f0ad80 P a3466cf67681447890434b1fc93bba03: Bootstrap starting.
I20260928 14:56:22.695200 1722 tablet_bootstrap.cc:492] T 19571dfe91ca4ad49671b8b000f0ad80 P 85d458ba5c774fca927a3fcd6362fb71: Bootstrap starting.
I20260928 14:56:22.695230 1720 tablet_bootstrap.cc:654] T 19571dfe91ca4ad49671b8b000f0ad80 P c9fce3001c8a4244b058751cace83dd0: Neither blocks nor log segments found. Creating new log.
I20260928 14:56:22.696759 1721 tablet_bootstrap.cc:654] T 19571dfe91ca4ad49671b8b000f0ad80 P a3466cf67681447890434b1fc93bba03: Neither blocks nor log segments found. Creating new log.
I20260928 14:56:22.697214 1722 tablet_bootstrap.cc:654] T 19571dfe91ca4ad49671b8b000f0ad80 P 85d458ba5c774fca927a3fcd6362fb71: Neither blocks nor log segments found. Creating new log.
I20260928 14:56:22.699815 1722 tablet_bootstrap.cc:492] T 19571dfe91ca4ad49671b8b000f0ad80 P 85d458ba5c774fca927a3fcd6362fb71: No bootstrap required, opened a new log
I20260928 14:56:22.700057 1722 ts_tablet_manager.cc:1397] T 19571dfe91ca4ad49671b8b000f0ad80 P 85d458ba5c774fca927a3fcd6362fb71: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260928 14:56:22.700340 1721 tablet_bootstrap.cc:492] T 19571dfe91ca4ad49671b8b000f0ad80 P a3466cf67681447890434b1fc93bba03: No bootstrap required, opened a new log
I20260928 14:56:22.700474 1721 ts_tablet_manager.cc:1397] T 19571dfe91ca4ad49671b8b000f0ad80 P a3466cf67681447890434b1fc93bba03: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 14:56:22.701043 1720 tablet_bootstrap.cc:492] T 19571dfe91ca4ad49671b8b000f0ad80 P c9fce3001c8a4244b058751cace83dd0: No bootstrap required, opened a new log
I20260928 14:56:22.700878 1722 raft_consensus.cc:348] T 19571dfe91ca4ad49671b8b000f0ad80 P 85d458ba5c774fca927a3fcd6362fb71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 } } peers { permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 } } peers { permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 } }
I20260928 14:56:22.701215 1722 raft_consensus.cc:374] T 19571dfe91ca4ad49671b8b000f0ad80 P 85d458ba5c774fca927a3fcd6362fb71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:56:22.701310 1720 ts_tablet_manager.cc:1397] T 19571dfe91ca4ad49671b8b000f0ad80 P c9fce3001c8a4244b058751cace83dd0: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.002s
I20260928 14:56:22.701387 1722 raft_consensus.cc:729] T 19571dfe91ca4ad49671b8b000f0ad80 P 85d458ba5c774fca927a3fcd6362fb71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85d458ba5c774fca927a3fcd6362fb71, State: Initialized, Role: FOLLOWER
I20260928 14:56:22.701201 1721 raft_consensus.cc:348] T 19571dfe91ca4ad49671b8b000f0ad80 P a3466cf67681447890434b1fc93bba03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 } } peers { permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 } } peers { permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 } }
I20260928 14:56:22.701493 1721 raft_consensus.cc:374] T 19571dfe91ca4ad49671b8b000f0ad80 P a3466cf67681447890434b1fc93bba03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:56:22.701591 1721 raft_consensus.cc:729] T 19571dfe91ca4ad49671b8b000f0ad80 P a3466cf67681447890434b1fc93bba03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3466cf67681447890434b1fc93bba03, State: Initialized, Role: FOLLOWER
I20260928 14:56:22.701632 1722 consensus_queue.cc:260] T 19571dfe91ca4ad49671b8b000f0ad80 P 85d458ba5c774fca927a3fcd6362fb71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 } } peers { permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 } } peers { permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 } }
I20260928 14:56:22.701808 1721 consensus_queue.cc:260] T 19571dfe91ca4ad49671b8b000f0ad80 P a3466cf67681447890434b1fc93bba03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 } } peers { permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 } } peers { permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 } }
I20260928 14:56:22.702227 1722 ts_tablet_manager.cc:1428] T 19571dfe91ca4ad49671b8b000f0ad80 P 85d458ba5c774fca927a3fcd6362fb71: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 14:56:22.702349 1721 ts_tablet_manager.cc:1428] T 19571dfe91ca4ad49671b8b000f0ad80 P a3466cf67681447890434b1fc93bba03: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260928 14:56:22.702189 1720 raft_consensus.cc:348] T 19571dfe91ca4ad49671b8b000f0ad80 P c9fce3001c8a4244b058751cace83dd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 } } peers { permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 } } peers { permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 } }
I20260928 14:56:22.702448 1720 raft_consensus.cc:374] T 19571dfe91ca4ad49671b8b000f0ad80 P c9fce3001c8a4244b058751cace83dd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:56:22.702544 1720 raft_consensus.cc:729] T 19571dfe91ca4ad49671b8b000f0ad80 P c9fce3001c8a4244b058751cace83dd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9fce3001c8a4244b058751cace83dd0, State: Initialized, Role: FOLLOWER
I20260928 14:56:22.702626 1728 tablet_replica.cc:439] T 19571dfe91ca4ad49671b8b000f0ad80 P a3466cf67681447890434b1fc93bba03: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 } } peers { permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 } } peers { permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 } } }
I20260928 14:56:22.702764 1720 consensus_queue.cc:260] T 19571dfe91ca4ad49671b8b000f0ad80 P c9fce3001c8a4244b058751cace83dd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 } } peers { permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 } } peers { permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 } }
I20260928 14:56:22.702937 1728 tablet_replica.cc:442] T 19571dfe91ca4ad49671b8b000f0ad80 P a3466cf67681447890434b1fc93bba03: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:56:22.703392 1726 tablet_replica.cc:439] T 19571dfe91ca4ad49671b8b000f0ad80 P c9fce3001c8a4244b058751cace83dd0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 } } peers { permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 } } peers { permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 } } }
I20260928 14:56:22.703739 1726 tablet_replica.cc:442] T 19571dfe91ca4ad49671b8b000f0ad80 P c9fce3001c8a4244b058751cace83dd0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:56:22.704855 1720 ts_tablet_manager.cc:1428] T 19571dfe91ca4ad49671b8b000f0ad80 P c9fce3001c8a4244b058751cace83dd0: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260928 14:56:22.705097 1727 tablet_replica.cc:439] T 19571dfe91ca4ad49671b8b000f0ad80 P 85d458ba5c774fca927a3fcd6362fb71: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 } } peers { permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 } } peers { permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 } } }
I20260928 14:56:22.705401 1727 tablet_replica.cc:442] T 19571dfe91ca4ad49671b8b000f0ad80 P 85d458ba5c774fca927a3fcd6362fb71: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:56:22.836781 1726 raft_consensus.cc:482] T 19571dfe91ca4ad49671b8b000f0ad80 P c9fce3001c8a4244b058751cace83dd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:56:22.837090 1726 raft_consensus.cc:504] T 19571dfe91ca4ad49671b8b000f0ad80 P c9fce3001c8a4244b058751cace83dd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 } } peers { permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 } } peers { permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 } }
I20260928 14:56:22.838631 1726 leader_election.cc:288] T 19571dfe91ca4ad49671b8b000f0ad80 P c9fce3001c8a4244b058751cace83dd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85d458ba5c774fca927a3fcd6362fb71 (127.31.40.2:39879), a3466cf67681447890434b1fc93bba03 (127.31.40.3:33391)
I20260928 14:56:22.846817 1718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:22.839125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:59369 (local address 127.31.40.3:33391)
0928 14:56:22.839338 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:22.839345 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:22.840238 (+ 893us) server_negotiation.cc:407] Connection header received
0928 14:56:22.840435 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:22.840439 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:22.840499 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:22.840591 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:56:22.841176 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:22.841904 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:22.844396 (+ 2492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:22.845737 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:22.845858 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:22.845969 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:22.846376 (+ 407us) server_negotiation.cc:299] Negotiation successful
0928 14:56:22.846529 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260928 14:56:22.846905 1715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:22.839072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:45063 (local address 127.31.40.2:39879)
0928 14:56:22.839280 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:22.839286 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:22.840110 (+ 824us) server_negotiation.cc:407] Connection header received
0928 14:56:22.840339 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:22.840344 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:22.840404 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:22.840516 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:56:22.841124 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:22.841902 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:22.844363 (+ 2461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:22.845723 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:22.845859 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:22.845969 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:22.846492 (+ 523us) server_negotiation.cc:299] Negotiation successful
0928 14:56:22.846630 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 14:56:22.847234 1745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:22.839302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.3:33391 (local address 127.31.40.1:59369)
0928 14:56:22.840127 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:22.840141 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:22.840294 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:22.840605 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:22.840609 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:22.840637 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:56:22.841014 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:22.841022 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:22.842037 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:22.842045 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:22.844200 (+ 2155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:22.844216 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:22.845943 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:22.845951 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:22.846202 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:22.846244 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 14:56:22.846331 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 14:56:22.846456 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"mutex_wait_us":105,"thread_start_us":474,"threads_started":1}
I20260928 14:56:22.847280 1744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:22.839223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.2:39879 (local address 127.31.40.1:45063)
0928 14:56:22.839984 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:22.840006 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:22.840186 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:22.840546 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:22.840551 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:22.840575 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:56:22.840955 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:22.840965 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:22.842038 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:22.842046 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:22.844201 (+ 2155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:22.844216 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:22.846159 (+ 1943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:22.846166 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:22.846303 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:22.846353 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 14:56:22.846449 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 14:56:22.846579 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"mutex_wait_us":133,"thread_start_us":421,"threads_started":1}
I20260928 14:56:22.847920 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19571dfe91ca4ad49671b8b000f0ad80" candidate_uuid: "c9fce3001c8a4244b058751cace83dd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3466cf67681447890434b1fc93bba03" is_pre_election: true
I20260928 14:56:22.847940 1511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19571dfe91ca4ad49671b8b000f0ad80" candidate_uuid: "c9fce3001c8a4244b058751cace83dd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85d458ba5c774fca927a3fcd6362fb71" is_pre_election: true
I20260928 14:56:22.848204 1656 raft_consensus.cc:2454] T 19571dfe91ca4ad49671b8b000f0ad80 P a3466cf67681447890434b1fc93bba03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9fce3001c8a4244b058751cace83dd0 in term 0.
I20260928 14:56:22.848232 1511 raft_consensus.cc:2454] T 19571dfe91ca4ad49671b8b000f0ad80 P 85d458ba5c774fca927a3fcd6362fb71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9fce3001c8a4244b058751cace83dd0 in term 0.
I20260928 14:56:22.848814 1301 leader_election.cc:302] T 19571dfe91ca4ad49671b8b000f0ad80 P c9fce3001c8a4244b058751cace83dd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3466cf67681447890434b1fc93bba03, c9fce3001c8a4244b058751cace83dd0; no voters:
I20260928 14:56:22.849331 1726 raft_consensus.cc:2781] T 19571dfe91ca4ad49671b8b000f0ad80 P c9fce3001c8a4244b058751cace83dd0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:56:22.849466 1726 raft_consensus.cc:482] T 19571dfe91ca4ad49671b8b000f0ad80 P c9fce3001c8a4244b058751cace83dd0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:56:22.849573 1726 raft_consensus.cc:3037] T 19571dfe91ca4ad49671b8b000f0ad80 P c9fce3001c8a4244b058751cace83dd0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:56:22.851683 1726 raft_consensus.cc:504] T 19571dfe91ca4ad49671b8b000f0ad80 P c9fce3001c8a4244b058751cace83dd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 } } peers { permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 } } peers { permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 } }
I20260928 14:56:22.852351 1726 leader_election.cc:288] T 19571dfe91ca4ad49671b8b000f0ad80 P c9fce3001c8a4244b058751cace83dd0 [CANDIDATE]: Term 1 election: Requested vote from peers 85d458ba5c774fca927a3fcd6362fb71 (127.31.40.2:39879), a3466cf67681447890434b1fc93bba03 (127.31.40.3:33391)
I20260928 14:56:22.852972 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19571dfe91ca4ad49671b8b000f0ad80" candidate_uuid: "c9fce3001c8a4244b058751cace83dd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3466cf67681447890434b1fc93bba03"
I20260928 14:56:22.852979 1511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19571dfe91ca4ad49671b8b000f0ad80" candidate_uuid: "c9fce3001c8a4244b058751cace83dd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85d458ba5c774fca927a3fcd6362fb71"
I20260928 14:56:22.853229 1656 raft_consensus.cc:3037] T 19571dfe91ca4ad49671b8b000f0ad80 P a3466cf67681447890434b1fc93bba03 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:56:22.853349 1511 raft_consensus.cc:3037] T 19571dfe91ca4ad49671b8b000f0ad80 P 85d458ba5c774fca927a3fcd6362fb71 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:56:22.855413 1656 raft_consensus.cc:2454] T 19571dfe91ca4ad49671b8b000f0ad80 P a3466cf67681447890434b1fc93bba03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9fce3001c8a4244b058751cace83dd0 in term 1.
I20260928 14:56:22.855671 1511 raft_consensus.cc:2454] T 19571dfe91ca4ad49671b8b000f0ad80 P 85d458ba5c774fca927a3fcd6362fb71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9fce3001c8a4244b058751cace83dd0 in term 1.
I20260928 14:56:22.855921 1301 leader_election.cc:302] T 19571dfe91ca4ad49671b8b000f0ad80 P c9fce3001c8a4244b058751cace83dd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3466cf67681447890434b1fc93bba03, c9fce3001c8a4244b058751cace83dd0; no voters:
I20260928 14:56:22.856261 1726 raft_consensus.cc:2781] T 19571dfe91ca4ad49671b8b000f0ad80 P c9fce3001c8a4244b058751cace83dd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:56:22.856694 1726 raft_consensus.cc:686] T 19571dfe91ca4ad49671b8b000f0ad80 P c9fce3001c8a4244b058751cace83dd0 [term 1 LEADER]: Becoming Leader. State: Replica: c9fce3001c8a4244b058751cace83dd0, State: Running, Role: LEADER
I20260928 14:56:22.857198 1726 consensus_queue.cc:237] T 19571dfe91ca4ad49671b8b000f0ad80 P c9fce3001c8a4244b058751cace83dd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 } } peers { permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 } } peers { permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 } }
I20260928 14:56:22.860811 1746 tablet_replica.cc:439] T 19571dfe91ca4ad49671b8b000f0ad80 P c9fce3001c8a4244b058751cace83dd0: TxnStatusTablet state changed. Reason: New leader c9fce3001c8a4244b058751cace83dd0. Latest consensus state: current_term: 1 leader_uuid: "c9fce3001c8a4244b058751cace83dd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 } } peers { permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 } } peers { permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 } } }
I20260928 14:56:22.861235 1746 tablet_replica.cc:442] T 19571dfe91ca4ad49671b8b000f0ad80 P c9fce3001c8a4244b058751cace83dd0: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:56:22.862095 1748 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 14:56:22.862355 1748 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 14:56:22.862664 1209 catalog_manager.cc:5495] T 19571dfe91ca4ad49671b8b000f0ad80 P c9fce3001c8a4244b058751cace83dd0 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9fce3001c8a4244b058751cace83dd0 (127.31.40.1). New cstate: current_term: 1 leader_uuid: "c9fce3001c8a4244b058751cace83dd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 } health_report { overall_health: UNKNOWN } } }
I20260928 14:56:22.908651 1714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:22.901858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48448 (local address 127.31.40.1:36267)
0928 14:56:22.902109 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:22.902116 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:22.902263 (+ 147us) server_negotiation.cc:407] Connection header received
0928 14:56:22.902631 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:22.902636 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:22.902771 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:22.902957 (+ 186us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:56:22.903355 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:22.904475 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:22.905917 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:22.906379 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:22.906540 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:22.907049 (+ 509us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:56:22.907926 (+ 877us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:56:22.908100 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:22.908199 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 14:56:22.908315 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260928 14:56:22.908741 1740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:22.901785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:36267 (local address 127.0.0.1:48448)
0928 14:56:22.902165 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:22.902184 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:22.902363 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:22.902978 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:22.902983 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:22.902996 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:56:22.903211 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:22.903220 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:22.904641 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:22.904647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:22.905729 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:22.905743 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:22.906583 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:22.906591 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:22.906756 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:22.906818 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:56:22.908046 (+ 1228us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:56:22.908082 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 14:56:22.908179 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 14:56:22.908346 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 14:56:22.918442 1656 raft_consensus.cc:1261] T 19571dfe91ca4ad49671b8b000f0ad80 P a3466cf67681447890434b1fc93bba03 [term 1 FOLLOWER]: Refusing update from remote peer c9fce3001c8a4244b058751cace83dd0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:56:22.918413 1511 raft_consensus.cc:1261] T 19571dfe91ca4ad49671b8b000f0ad80 P 85d458ba5c774fca927a3fcd6362fb71 [term 1 FOLLOWER]: Refusing update from remote peer c9fce3001c8a4244b058751cace83dd0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:56:22.919381 1726 consensus_queue.cc:1035] T 19571dfe91ca4ad49671b8b000f0ad80 P c9fce3001c8a4244b058751cace83dd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:56:22.919745 1746 consensus_queue.cc:1035] T 19571dfe91ca4ad49671b8b000f0ad80 P c9fce3001c8a4244b058751cace83dd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:56:22.935333 1728 tablet_replica.cc:439] T 19571dfe91ca4ad49671b8b000f0ad80 P a3466cf67681447890434b1fc93bba03: TxnStatusTablet state changed. Reason: New leader c9fce3001c8a4244b058751cace83dd0. Latest consensus state: current_term: 1 leader_uuid: "c9fce3001c8a4244b058751cace83dd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 } } peers { permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 } } peers { permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 } } }
I20260928 14:56:22.935755 1728 tablet_replica.cc:442] T 19571dfe91ca4ad49671b8b000f0ad80 P a3466cf67681447890434b1fc93bba03: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:56:22.937067 1727 tablet_replica.cc:439] T 19571dfe91ca4ad49671b8b000f0ad80 P 85d458ba5c774fca927a3fcd6362fb71: TxnStatusTablet state changed. Reason: New leader c9fce3001c8a4244b058751cace83dd0. Latest consensus state: current_term: 1 leader_uuid: "c9fce3001c8a4244b058751cace83dd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 } } peers { permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 } } peers { permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 } } }
I20260928 14:56:22.937425 1727 tablet_replica.cc:442] T 19571dfe91ca4ad49671b8b000f0ad80 P 85d458ba5c774fca927a3fcd6362fb71: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:56:22.941066 1753 mvcc.cc:205] Tried to move back new op lower bound from 7334327840422166528 to 7334327840187015168. Current Snapshot: MvccSnapshot[applied={T|T < 7334327840422166528}]
I20260928 14:56:22.941633 1727 tablet_replica.cc:439] T 19571dfe91ca4ad49671b8b000f0ad80 P 85d458ba5c774fca927a3fcd6362fb71: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c9fce3001c8a4244b058751cace83dd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 } } peers { permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 } } peers { permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 } } }
I20260928 14:56:22.942009 1727 tablet_replica.cc:442] T 19571dfe91ca4ad49671b8b000f0ad80 P 85d458ba5c774fca927a3fcd6362fb71: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:56:22.942204 1751 mvcc.cc:205] Tried to move back new op lower bound from 7334327840422166528 to 7334327840187015168. Current Snapshot: MvccSnapshot[applied={T|T < 7334327840422166528}]
I20260928 14:56:22.946496 1728 tablet_replica.cc:439] T 19571dfe91ca4ad49671b8b000f0ad80 P a3466cf67681447890434b1fc93bba03: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c9fce3001c8a4244b058751cace83dd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 } } peers { permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 } } peers { permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 } } }
I20260928 14:56:22.946828 1728 tablet_replica.cc:442] T 19571dfe91ca4ad49671b8b000f0ad80 P a3466cf67681447890434b1fc93bba03: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:56:22.950083 1726 tablet_replica.cc:439] T 19571dfe91ca4ad49671b8b000f0ad80 P c9fce3001c8a4244b058751cace83dd0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c9fce3001c8a4244b058751cace83dd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 } } peers { permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 } } peers { permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 } } }
I20260928 14:56:22.950397 1726 tablet_replica.cc:442] T 19571dfe91ca4ad49671b8b000f0ad80 P c9fce3001c8a4244b058751cace83dd0: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:56:22.950645 1756 tablet_replica.cc:439] T 19571dfe91ca4ad49671b8b000f0ad80 P c9fce3001c8a4244b058751cace83dd0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c9fce3001c8a4244b058751cace83dd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 } } peers { permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 } } peers { permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 } } }
I20260928 14:56:22.951073 1756 tablet_replica.cc:442] T 19571dfe91ca4ad49671b8b000f0ad80 P c9fce3001c8a4244b058751cace83dd0: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:56:22.974912 1715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:22.958827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43442 (local address 127.31.40.2:39879)
0928 14:56:22.959036 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:22.959042 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:22.967909 (+ 8867us) server_negotiation.cc:407] Connection header received
0928 14:56:22.968190 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:22.968197 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:22.968327 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:22.968526 (+ 199us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:56:22.969020 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:22.970308 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:22.971984 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:22.972754 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:22.972926 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:22.973330 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:56:22.974188 (+ 858us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:56:22.974352 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:22.974481 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 14:56:22.974582 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 14:56:22.997686 1765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:22.987988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:36267 (local address 127.31.40.2:50871)
0928 14:56:22.991782 (+ 3794us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:22.991817 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:22.992040 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:22.992428 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:22.992432 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:22.992442 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:56:22.992807 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:22.992817 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:22.994260 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:22.994264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:22.995350 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:22.995362 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:22.996464 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:22.996470 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:22.996591 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:22.996667 (+ 76us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:56:22.997121 (+ 454us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:56:22.997130 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 14:56:22.997198 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 14:56:22.997343 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3622,"thread_start_us":3429,"threads_started":1}
I20260928 14:56:22.997673 1714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:22.988093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:50871 (local address 127.31.40.1:36267)
0928 14:56:22.988266 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:22.988271 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:22.991990 (+ 3719us) server_negotiation.cc:407] Connection header received
0928 14:56:22.992175 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:22.992181 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:22.992271 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:22.992377 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:56:22.992974 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:22.994129 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:22.995493 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:22.996266 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:22.996393 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:22.996868 (+ 475us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:56:22.997013 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:56:22.997113 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:22.997247 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 14:56:22.997343 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 14:56:23.008476 1656 raft_consensus.cc:1261] T 6732be1dc9f94e398737050018dfda7b P a3466cf67681447890434b1fc93bba03 [term 1 FOLLOWER]: Refusing update from remote peer 85d458ba5c774fca927a3fcd6362fb71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:56:23.008864 1371 raft_consensus.cc:1261] T 6732be1dc9f94e398737050018dfda7b P c9fce3001c8a4244b058751cace83dd0 [term 1 FOLLOWER]: Refusing update from remote peer 85d458ba5c774fca927a3fcd6362fb71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:56:23.009682 1733 consensus_queue.cc:1035] T 6732be1dc9f94e398737050018dfda7b P 85d458ba5c774fca927a3fcd6362fb71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:56:23.010062 1727 consensus_queue.cc:1035] T 6732be1dc9f94e398737050018dfda7b P 85d458ba5c774fca927a3fcd6362fb71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:56:23.047617 1715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:23.040452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:48853 (local address 127.31.40.2:39879)
0928 14:56:23.040651 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:23.040659 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:23.041289 (+ 630us) server_negotiation.cc:407] Connection header received
0928 14:56:23.041548 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:23.041553 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:23.041650 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:23.041749 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:56:23.042396 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:23.043554 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:23.045402 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:23.046159 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:23.046289 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:23.046777 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:56:23.046924 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:56:23.047180 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:23.047319 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 14:56:23.047386 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 14:56:23.048051 1769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:23.040414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.2:39879 (local address 127.31.40.1:48853)
0928 14:56:23.041193 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:23.041211 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:23.041393 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:23.041762 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:23.041767 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:23.041778 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:56:23.042232 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:23.042245 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:23.043697 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:23.043701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:23.045239 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:23.045271 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:23.046343 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:23.046350 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:23.046514 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:23.046589 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:56:23.047188 (+ 599us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:56:23.047201 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 14:56:23.047297 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 14:56:23.047451 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":399,"threads_started":1}
I20260928 14:56:23.098295 1210 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:56:23.098589 1210 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:56:23.102679 1210 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b9e26c8ffca249b9a82a4ec88f68f23b: Sending DeleteTablet for 3 replicas of tablet 6732be1dc9f94e398737050018dfda7b
I20260928 14:56:23.103945 1491 tablet_service.cc:1514] Processing DeleteTablet for tablet 6732be1dc9f94e398737050018dfda7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:56:23 UTC) from {username='kudu', principal='kudu/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:43434
I20260928 14:56:23.103962 1351 tablet_service.cc:1514] Processing DeleteTablet for tablet 6732be1dc9f94e398737050018dfda7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:56:23 UTC) from {username='kudu', principal='kudu/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:48438
I20260928 14:56:23.108961 1631 tablet_service.cc:1514] Processing DeleteTablet for tablet 6732be1dc9f94e398737050018dfda7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:56:23 UTC) from {username='kudu', principal='kudu/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:49986
I20260928 14:56:23.117378 1773 tablet_replica.cc:330] T 6732be1dc9f94e398737050018dfda7b P 85d458ba5c774fca927a3fcd6362fb71: stopping tablet replica
I20260928 14:56:23.117915 1773 raft_consensus.cc:2229] T 6732be1dc9f94e398737050018dfda7b P 85d458ba5c774fca927a3fcd6362fb71 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:56:23.118607 1773 raft_consensus.cc:2258] T 6732be1dc9f94e398737050018dfda7b P 85d458ba5c774fca927a3fcd6362fb71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:56:23.119004 1210 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43832:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:56:23.120690 1210 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:56:23.121145 1773 ts_tablet_manager.cc:1905] T 6732be1dc9f94e398737050018dfda7b P 85d458ba5c774fca927a3fcd6362fb71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:56:23.129859 1773 ts_tablet_manager.cc:1918] T 6732be1dc9f94e398737050018dfda7b P 85d458ba5c774fca927a3fcd6362fb71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:56:23.130074 1773 log.cc:1196] T 6732be1dc9f94e398737050018dfda7b P 85d458ba5c774fca927a3fcd6362fb71: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-1/wal/wals/6732be1dc9f94e398737050018dfda7b
I20260928 14:56:23.130676 1773 ts_tablet_manager.cc:1939] T 6732be1dc9f94e398737050018dfda7b P 85d458ba5c774fca927a3fcd6362fb71: Deleting consensus metadata
I20260928 14:56:23.132334 1774 tablet_replica.cc:330] T 6732be1dc9f94e398737050018dfda7b P c9fce3001c8a4244b058751cace83dd0: stopping tablet replica
I20260928 14:56:23.132762 1195 catalog_manager.cc:4841] TS 85d458ba5c774fca927a3fcd6362fb71 (127.31.40.2:39879): tablet 6732be1dc9f94e398737050018dfda7b (table test-table [id=d7d08e383fc94168bd9a40a5bc5ceb30]) successfully deleted
I20260928 14:56:23.132845 1774 raft_consensus.cc:2229] T 6732be1dc9f94e398737050018dfda7b P c9fce3001c8a4244b058751cace83dd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:56:23.133252 1774 raft_consensus.cc:2258] T 6732be1dc9f94e398737050018dfda7b P c9fce3001c8a4244b058751cace83dd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:56:23.134742 1491 tablet_service.cc:1467] Processing CreateTablet for tablet aa4581f30e4c432e914472b4cc005c4e (DEFAULT_TABLE table=test-table [id=ba30c56636bb4db19ea87ba1f747cd6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:56:23.135195 1491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa4581f30e4c432e914472b4cc005c4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:56:23.135581 1630 tablet_service.cc:1467] Processing CreateTablet for tablet aa4581f30e4c432e914472b4cc005c4e (DEFAULT_TABLE table=test-table [id=ba30c56636bb4db19ea87ba1f747cd6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:56:23.136193 1630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa4581f30e4c432e914472b4cc005c4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:56:23.137413 1350 tablet_service.cc:1467] Processing CreateTablet for tablet aa4581f30e4c432e914472b4cc005c4e (DEFAULT_TABLE table=test-table [id=ba30c56636bb4db19ea87ba1f747cd6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:56:23.137887 1350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa4581f30e4c432e914472b4cc005c4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:56:23.140889 1775 tablet_replica.cc:330] T 6732be1dc9f94e398737050018dfda7b P a3466cf67681447890434b1fc93bba03: stopping tablet replica
I20260928 14:56:23.141907 1722 tablet_bootstrap.cc:492] T aa4581f30e4c432e914472b4cc005c4e P 85d458ba5c774fca927a3fcd6362fb71: Bootstrap starting.
I20260928 14:56:23.143829 1721 tablet_bootstrap.cc:492] T aa4581f30e4c432e914472b4cc005c4e P a3466cf67681447890434b1fc93bba03: Bootstrap starting.
I20260928 14:56:23.143908 1722 tablet_bootstrap.cc:654] T aa4581f30e4c432e914472b4cc005c4e P 85d458ba5c774fca927a3fcd6362fb71: Neither blocks nor log segments found. Creating new log.
I20260928 14:56:23.144268 1774 ts_tablet_manager.cc:1905] T 6732be1dc9f94e398737050018dfda7b P c9fce3001c8a4244b058751cace83dd0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:56:23.145890 1721 tablet_bootstrap.cc:654] T aa4581f30e4c432e914472b4cc005c4e P a3466cf67681447890434b1fc93bba03: Neither blocks nor log segments found. Creating new log.
I20260928 14:56:23.149981 1720 tablet_bootstrap.cc:492] T aa4581f30e4c432e914472b4cc005c4e P c9fce3001c8a4244b058751cace83dd0: Bootstrap starting.
I20260928 14:56:23.141558 1775 raft_consensus.cc:2229] T 6732be1dc9f94e398737050018dfda7b P a3466cf67681447890434b1fc93bba03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:56:23.151011 1775 raft_consensus.cc:2258] T 6732be1dc9f94e398737050018dfda7b P a3466cf67681447890434b1fc93bba03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:56:23.151894 1720 tablet_bootstrap.cc:654] T aa4581f30e4c432e914472b4cc005c4e P c9fce3001c8a4244b058751cace83dd0: Neither blocks nor log segments found. Creating new log.
I20260928 14:56:23.156107 1775 ts_tablet_manager.cc:1905] T 6732be1dc9f94e398737050018dfda7b P a3466cf67681447890434b1fc93bba03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:56:23.162189 1720 tablet_bootstrap.cc:492] T aa4581f30e4c432e914472b4cc005c4e P c9fce3001c8a4244b058751cace83dd0: No bootstrap required, opened a new log
I20260928 14:56:23.162405 1720 ts_tablet_manager.cc:1397] T aa4581f30e4c432e914472b4cc005c4e P c9fce3001c8a4244b058751cace83dd0: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.005s
I20260928 14:56:23.163241 1720 raft_consensus.cc:348] T aa4581f30e4c432e914472b4cc005c4e P c9fce3001c8a4244b058751cace83dd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 } } peers { permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 } } peers { permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 } }
I20260928 14:56:23.163529 1720 raft_consensus.cc:374] T aa4581f30e4c432e914472b4cc005c4e P c9fce3001c8a4244b058751cace83dd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:56:23.163618 1720 raft_consensus.cc:729] T aa4581f30e4c432e914472b4cc005c4e P c9fce3001c8a4244b058751cace83dd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9fce3001c8a4244b058751cace83dd0, State: Initialized, Role: FOLLOWER
I20260928 14:56:23.164011 1720 consensus_queue.cc:260] T aa4581f30e4c432e914472b4cc005c4e P c9fce3001c8a4244b058751cace83dd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 } } peers { permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 } } peers { permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 } }
I20260928 14:56:23.165838 1720 ts_tablet_manager.cc:1428] T aa4581f30e4c432e914472b4cc005c4e P c9fce3001c8a4244b058751cace83dd0: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260928 14:56:23.167030 1722 tablet_bootstrap.cc:492] T aa4581f30e4c432e914472b4cc005c4e P 85d458ba5c774fca927a3fcd6362fb71: No bootstrap required, opened a new log
I20260928 14:56:23.166363 1774 ts_tablet_manager.cc:1918] T 6732be1dc9f94e398737050018dfda7b P c9fce3001c8a4244b058751cace83dd0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:56:23.167315 1774 log.cc:1196] T 6732be1dc9f94e398737050018dfda7b P c9fce3001c8a4244b058751cace83dd0: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-0/wal/wals/6732be1dc9f94e398737050018dfda7b
I20260928 14:56:23.167335 1722 ts_tablet_manager.cc:1397] T aa4581f30e4c432e914472b4cc005c4e P 85d458ba5c774fca927a3fcd6362fb71: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.007s
I20260928 14:56:23.167975 1774 ts_tablet_manager.cc:1939] T 6732be1dc9f94e398737050018dfda7b P c9fce3001c8a4244b058751cace83dd0: Deleting consensus metadata
I20260928 14:56:23.168713 1722 raft_consensus.cc:348] T aa4581f30e4c432e914472b4cc005c4e P 85d458ba5c774fca927a3fcd6362fb71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 } } peers { permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 } } peers { permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 } }
I20260928 14:56:23.169939 1197 catalog_manager.cc:4841] TS c9fce3001c8a4244b058751cace83dd0 (127.31.40.1:36267): tablet 6732be1dc9f94e398737050018dfda7b (table test-table [id=d7d08e383fc94168bd9a40a5bc5ceb30]) successfully deleted
I20260928 14:56:23.168999 1722 raft_consensus.cc:374] T aa4581f30e4c432e914472b4cc005c4e P 85d458ba5c774fca927a3fcd6362fb71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:56:23.170667 1722 raft_consensus.cc:729] T aa4581f30e4c432e914472b4cc005c4e P 85d458ba5c774fca927a3fcd6362fb71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85d458ba5c774fca927a3fcd6362fb71, State: Initialized, Role: FOLLOWER
I20260928 14:56:23.168020 1721 tablet_bootstrap.cc:492] T aa4581f30e4c432e914472b4cc005c4e P a3466cf67681447890434b1fc93bba03: No bootstrap required, opened a new log
I20260928 14:56:23.171034 1722 consensus_queue.cc:260] T aa4581f30e4c432e914472b4cc005c4e P 85d458ba5c774fca927a3fcd6362fb71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 } } peers { permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 } } peers { permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 } }
I20260928 14:56:23.171784 1722 ts_tablet_manager.cc:1428] T aa4581f30e4c432e914472b4cc005c4e P 85d458ba5c774fca927a3fcd6362fb71: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260928 14:56:23.171382 1721 ts_tablet_manager.cc:1397] T aa4581f30e4c432e914472b4cc005c4e P a3466cf67681447890434b1fc93bba03: Time spent bootstrapping tablet: real 0.028s user 0.006s sys 0.004s
I20260928 14:56:23.177222 1721 raft_consensus.cc:348] T aa4581f30e4c432e914472b4cc005c4e P a3466cf67681447890434b1fc93bba03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 } } peers { permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 } } peers { permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 } }
I20260928 14:56:23.177597 1721 raft_consensus.cc:374] T aa4581f30e4c432e914472b4cc005c4e P a3466cf67681447890434b1fc93bba03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:56:23.177703 1721 raft_consensus.cc:729] T aa4581f30e4c432e914472b4cc005c4e P a3466cf67681447890434b1fc93bba03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3466cf67681447890434b1fc93bba03, State: Initialized, Role: FOLLOWER
I20260928 14:56:23.178144 1721 consensus_queue.cc:260] T aa4581f30e4c432e914472b4cc005c4e P a3466cf67681447890434b1fc93bba03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 } } peers { permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 } } peers { permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 } }
I20260928 14:56:23.178483 1728 raft_consensus.cc:482] T aa4581f30e4c432e914472b4cc005c4e P a3466cf67681447890434b1fc93bba03 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:56:23.178617 1728 raft_consensus.cc:504] T aa4581f30e4c432e914472b4cc005c4e P a3466cf67681447890434b1fc93bba03 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 } } peers { permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 } } peers { permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 } }
I20260928 14:56:23.179297 1728 leader_election.cc:288] T aa4581f30e4c432e914472b4cc005c4e P a3466cf67681447890434b1fc93bba03 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9fce3001c8a4244b058751cace83dd0 (127.31.40.1:36267), 85d458ba5c774fca927a3fcd6362fb71 (127.31.40.2:39879)
I20260928 14:56:23.180317 1775 ts_tablet_manager.cc:1918] T 6732be1dc9f94e398737050018dfda7b P a3466cf67681447890434b1fc93bba03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:56:23.180483 1775 log.cc:1196] T 6732be1dc9f94e398737050018dfda7b P a3466cf67681447890434b1fc93bba03: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-2/wal/wals/6732be1dc9f94e398737050018dfda7b
I20260928 14:56:23.180470 1371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa4581f30e4c432e914472b4cc005c4e" candidate_uuid: "a3466cf67681447890434b1fc93bba03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9fce3001c8a4244b058751cace83dd0" is_pre_election: true
I20260928 14:56:23.180765 1371 raft_consensus.cc:2454] T aa4581f30e4c432e914472b4cc005c4e P c9fce3001c8a4244b058751cace83dd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3466cf67681447890434b1fc93bba03 in term 0.
I20260928 14:56:23.181054 1775 ts_tablet_manager.cc:1939] T 6732be1dc9f94e398737050018dfda7b P a3466cf67681447890434b1fc93bba03: Deleting consensus metadata
I20260928 14:56:23.181174 1511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa4581f30e4c432e914472b4cc005c4e" candidate_uuid: "a3466cf67681447890434b1fc93bba03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85d458ba5c774fca927a3fcd6362fb71" is_pre_election: true
I20260928 14:56:23.181450 1511 raft_consensus.cc:2454] T aa4581f30e4c432e914472b4cc005c4e P 85d458ba5c774fca927a3fcd6362fb71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3466cf67681447890434b1fc93bba03 in term 0.
I20260928 14:56:23.181958 1581 leader_election.cc:302] T aa4581f30e4c432e914472b4cc005c4e P a3466cf67681447890434b1fc93bba03 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85d458ba5c774fca927a3fcd6362fb71, a3466cf67681447890434b1fc93bba03; no voters:
I20260928 14:56:23.182710 1728 raft_consensus.cc:2781] T aa4581f30e4c432e914472b4cc005c4e P a3466cf67681447890434b1fc93bba03 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:56:23.182850 1728 raft_consensus.cc:482] T aa4581f30e4c432e914472b4cc005c4e P a3466cf67681447890434b1fc93bba03 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:56:23.182941 1728 raft_consensus.cc:3037] T aa4581f30e4c432e914472b4cc005c4e P a3466cf67681447890434b1fc93bba03 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:56:23.184295 1196 catalog_manager.cc:4841] TS a3466cf67681447890434b1fc93bba03 (127.31.40.3:33391): tablet 6732be1dc9f94e398737050018dfda7b (table test-table [id=d7d08e383fc94168bd9a40a5bc5ceb30]) successfully deleted
I20260928 14:56:23.186214 1728 raft_consensus.cc:504] T aa4581f30e4c432e914472b4cc005c4e P a3466cf67681447890434b1fc93bba03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 } } peers { permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 } } peers { permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 } }
I20260928 14:56:23.187477 1511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa4581f30e4c432e914472b4cc005c4e" candidate_uuid: "a3466cf67681447890434b1fc93bba03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85d458ba5c774fca927a3fcd6362fb71"
I20260928 14:56:23.187758 1511 raft_consensus.cc:3037] T aa4581f30e4c432e914472b4cc005c4e P 85d458ba5c774fca927a3fcd6362fb71 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:56:23.190995 1511 raft_consensus.cc:2454] T aa4581f30e4c432e914472b4cc005c4e P 85d458ba5c774fca927a3fcd6362fb71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3466cf67681447890434b1fc93bba03 in term 1.
I20260928 14:56:23.191536 1581 leader_election.cc:302] T aa4581f30e4c432e914472b4cc005c4e P a3466cf67681447890434b1fc93bba03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85d458ba5c774fca927a3fcd6362fb71, a3466cf67681447890434b1fc93bba03; no voters:
I20260928 14:56:23.191928 1371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa4581f30e4c432e914472b4cc005c4e" candidate_uuid: "a3466cf67681447890434b1fc93bba03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9fce3001c8a4244b058751cace83dd0"
I20260928 14:56:23.192169 1371 raft_consensus.cc:3037] T aa4581f30e4c432e914472b4cc005c4e P c9fce3001c8a4244b058751cace83dd0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:56:23.195585 1371 raft_consensus.cc:2454] T aa4581f30e4c432e914472b4cc005c4e P c9fce3001c8a4244b058751cace83dd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3466cf67681447890434b1fc93bba03 in term 1.
I20260928 14:56:23.196463 1728 leader_election.cc:288] T aa4581f30e4c432e914472b4cc005c4e P a3466cf67681447890434b1fc93bba03 [CANDIDATE]: Term 1 election: Requested vote from peers c9fce3001c8a4244b058751cace83dd0 (127.31.40.1:36267), 85d458ba5c774fca927a3fcd6362fb71 (127.31.40.2:39879)
I20260928 14:56:23.196750 1728 raft_consensus.cc:2781] T aa4581f30e4c432e914472b4cc005c4e P a3466cf67681447890434b1fc93bba03 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:56:23.196949 1728 raft_consensus.cc:686] T aa4581f30e4c432e914472b4cc005c4e P a3466cf67681447890434b1fc93bba03 [term 1 LEADER]: Becoming Leader. State: Replica: a3466cf67681447890434b1fc93bba03, State: Running, Role: LEADER
I20260928 14:56:23.197496 1728 consensus_queue.cc:237] T aa4581f30e4c432e914472b4cc005c4e P a3466cf67681447890434b1fc93bba03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 } } peers { permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 } } peers { permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 } }
I20260928 14:56:23.202986 1721 ts_tablet_manager.cc:1428] T aa4581f30e4c432e914472b4cc005c4e P a3466cf67681447890434b1fc93bba03: Time spent starting tablet: real 0.027s user 0.003s sys 0.000s
I20260928 14:56:23.204227 1210 catalog_manager.cc:5495] T aa4581f30e4c432e914472b4cc005c4e P a3466cf67681447890434b1fc93bba03 reported cstate change: term changed from 0 to 1, leader changed from <none> to a3466cf67681447890434b1fc93bba03 (127.31.40.3). New cstate: current_term: 1 leader_uuid: "a3466cf67681447890434b1fc93bba03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 } health_report { overall_health: HEALTHY } } }
I20260928 14:56:23.247905 1718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:23.240169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49998 (local address 127.31.40.3:33391)
0928 14:56:23.240362 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:23.240368 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:23.240473 (+ 105us) server_negotiation.cc:407] Connection header received
0928 14:56:23.240716 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:23.240721 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:23.240844 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:23.241022 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:56:23.241430 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:23.242590 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:23.244742 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:23.245420 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:23.245567 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:23.246058 (+ 491us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:56:23.246891 (+ 833us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:56:23.247044 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:23.247271 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 14:56:23.247368 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 14:56:23.272764 1714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:23.261466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:47329 (local address 127.31.40.1:36267)
0928 14:56:23.261653 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:23.261659 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:23.266038 (+ 4379us) server_negotiation.cc:407] Connection header received
0928 14:56:23.266249 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:23.266254 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:23.266347 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:23.266470 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:56:23.267101 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:23.268339 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:23.270396 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:23.271073 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:23.271212 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:23.271799 (+ 587us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:56:23.272013 (+ 214us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:56:23.272109 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:23.272269 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 14:56:23.272360 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 14:56:23.272827 1783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:23.261435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:36267 (local address 127.31.40.3:47329)
0928 14:56:23.265900 (+ 4465us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:23.265926 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:23.266097 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:23.266547 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:23.266552 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:23.266566 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:56:23.266900 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:23.266911 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:23.268485 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:23.268491 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:23.270235 (+ 1744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:23.270252 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:23.271245 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:23.271253 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:23.271471 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:23.271553 (+ 82us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:56:23.272133 (+ 580us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:56:23.272145 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 14:56:23.272228 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 14:56:23.272362 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4254,"thread_start_us":4110,"threads_started":1}
I20260928 14:56:23.284827 1371 raft_consensus.cc:1261] T aa4581f30e4c432e914472b4cc005c4e P c9fce3001c8a4244b058751cace83dd0 [term 1 FOLLOWER]: Refusing update from remote peer a3466cf67681447890434b1fc93bba03: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:56:23.285755 1779 consensus_queue.cc:1035] T aa4581f30e4c432e914472b4cc005c4e P a3466cf67681447890434b1fc93bba03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:56:23.289557 1511 raft_consensus.cc:1261] T aa4581f30e4c432e914472b4cc005c4e P 85d458ba5c774fca927a3fcd6362fb71 [term 1 FOLLOWER]: Refusing update from remote peer a3466cf67681447890434b1fc93bba03: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:56:23.290452 1779 consensus_queue.cc:1035] T aa4581f30e4c432e914472b4cc005c4e P a3466cf67681447890434b1fc93bba03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:56:23.312500 1752 mvcc.cc:205] Tried to move back new op lower bound from 7334327841923141632 to 7334327841585586176. Current Snapshot: MvccSnapshot[applied={T|T < 7334327841923141632}]
I20260928 14:56:23.356788 1769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:23.349031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.3:33391 (local address 127.31.40.1:47139)
0928 14:56:23.349417 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:23.349437 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:23.349553 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:23.350046 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:23.350051 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:23.350064 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:56:23.350291 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:23.350303 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:23.352385 (+ 2082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:23.352391 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:23.354135 (+ 1744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:23.354153 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:23.355392 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:23.355398 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:23.355610 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:23.355664 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:56:23.356172 (+ 508us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:56:23.356189 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 14:56:23.356293 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 14:56:23.356457 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233}
I20260928 14:56:23.357390 1718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:23.349548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:47139 (local address 127.31.40.3:33391)
0928 14:56:23.349715 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:23.349721 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:23.349742 (+ 21us) server_negotiation.cc:407] Connection header received
0928 14:56:23.349813 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:23.349818 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:23.349909 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:23.350004 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:56:23.350620 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:23.351831 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:23.354299 (+ 2468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:23.355215 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:23.355351 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:23.355853 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:56:23.356020 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:56:23.356113 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:23.357038 (+ 925us) server_negotiation.cc:299] Negotiation successful
0928 14:56:23.357120 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260928 14:56:23.359344 1752 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 14:56:25.006453 1209 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:56:25.006698 1209 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:56:25.011293 1209 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b9e26c8ffca249b9a82a4ec88f68f23b: Sending DeleteTablet for 3 replicas of tablet aa4581f30e4c432e914472b4cc005c4e
I20260928 14:56:25.012303 1350 tablet_service.cc:1514] Processing DeleteTablet for tablet aa4581f30e4c432e914472b4cc005c4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:56:25 UTC) from {username='kudu', principal='kudu/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:48438
I20260928 14:56:25.012351 1491 tablet_service.cc:1514] Processing DeleteTablet for tablet aa4581f30e4c432e914472b4cc005c4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:56:25 UTC) from {username='kudu', principal='kudu/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:43434
I20260928 14:56:25.012589 1630 tablet_service.cc:1514] Processing DeleteTablet for tablet aa4581f30e4c432e914472b4cc005c4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:56:25 UTC) from {username='kudu', principal='kudu/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:49986
I20260928 14:56:25.024531 1814 tablet_replica.cc:330] T aa4581f30e4c432e914472b4cc005c4e P 85d458ba5c774fca927a3fcd6362fb71: stopping tablet replica
I20260928 14:56:25.024883 1814 raft_consensus.cc:2229] T aa4581f30e4c432e914472b4cc005c4e P 85d458ba5c774fca927a3fcd6362fb71 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:56:25.025220 1814 raft_consensus.cc:2258] T aa4581f30e4c432e914472b4cc005c4e P 85d458ba5c774fca927a3fcd6362fb71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:56:25.025944 1813 tablet_replica.cc:330] T aa4581f30e4c432e914472b4cc005c4e P c9fce3001c8a4244b058751cace83dd0: stopping tablet replica
I20260928 14:56:25.026165 1815 tablet_replica.cc:330] T aa4581f30e4c432e914472b4cc005c4e P a3466cf67681447890434b1fc93bba03: stopping tablet replica
I20260928 14:56:25.026156 1813 raft_consensus.cc:2229] T aa4581f30e4c432e914472b4cc005c4e P c9fce3001c8a4244b058751cace83dd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:56:25.026381 1815 raft_consensus.cc:2229] T aa4581f30e4c432e914472b4cc005c4e P a3466cf67681447890434b1fc93bba03 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:56:25.026420 1813 raft_consensus.cc:2258] T aa4581f30e4c432e914472b4cc005c4e P c9fce3001c8a4244b058751cace83dd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:56:25.026789 1814 ts_tablet_manager.cc:1905] T aa4581f30e4c432e914472b4cc005c4e P 85d458ba5c774fca927a3fcd6362fb71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:56:25.026993 1815 raft_consensus.cc:2258] T aa4581f30e4c432e914472b4cc005c4e P a3466cf67681447890434b1fc93bba03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:56:25.027679 1813 ts_tablet_manager.cc:1905] T aa4581f30e4c432e914472b4cc005c4e P c9fce3001c8a4244b058751cace83dd0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:56:25.027993 1815 ts_tablet_manager.cc:1905] T aa4581f30e4c432e914472b4cc005c4e P a3466cf67681447890434b1fc93bba03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:56:25.032742 1814 ts_tablet_manager.cc:1918] T aa4581f30e4c432e914472b4cc005c4e P 85d458ba5c774fca927a3fcd6362fb71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 14:56:25.032902 1814 log.cc:1196] T aa4581f30e4c432e914472b4cc005c4e P 85d458ba5c774fca927a3fcd6362fb71: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-1/wal/wals/aa4581f30e4c432e914472b4cc005c4e
I20260928 14:56:25.033411 1814 ts_tablet_manager.cc:1939] T aa4581f30e4c432e914472b4cc005c4e P 85d458ba5c774fca927a3fcd6362fb71: Deleting consensus metadata
I20260928 14:56:25.034750 1815 ts_tablet_manager.cc:1918] T aa4581f30e4c432e914472b4cc005c4e P a3466cf67681447890434b1fc93bba03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 14:56:25.034749 1195 catalog_manager.cc:4841] TS 85d458ba5c774fca927a3fcd6362fb71 (127.31.40.2:39879): tablet aa4581f30e4c432e914472b4cc005c4e (table test-table [id=ba30c56636bb4db19ea87ba1f747cd6d]) successfully deleted
I20260928 14:56:25.034909 1815 log.cc:1196] T aa4581f30e4c432e914472b4cc005c4e P a3466cf67681447890434b1fc93bba03: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-2/wal/wals/aa4581f30e4c432e914472b4cc005c4e
I20260928 14:56:25.035194 1813 ts_tablet_manager.cc:1918] T aa4581f30e4c432e914472b4cc005c4e P c9fce3001c8a4244b058751cace83dd0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 14:56:25.035339 1813 log.cc:1196] T aa4581f30e4c432e914472b4cc005c4e P c9fce3001c8a4244b058751cace83dd0: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-0/wal/wals/aa4581f30e4c432e914472b4cc005c4e
I20260928 14:56:25.035379 1815 ts_tablet_manager.cc:1939] T aa4581f30e4c432e914472b4cc005c4e P a3466cf67681447890434b1fc93bba03: Deleting consensus metadata
I20260928 14:56:25.035813 1813 ts_tablet_manager.cc:1939] T aa4581f30e4c432e914472b4cc005c4e P c9fce3001c8a4244b058751cace83dd0: Deleting consensus metadata
I20260928 14:56:25.036684 1196 catalog_manager.cc:4841] TS a3466cf67681447890434b1fc93bba03 (127.31.40.3:33391): tablet aa4581f30e4c432e914472b4cc005c4e (table test-table [id=ba30c56636bb4db19ea87ba1f747cd6d]) successfully deleted
I20260928 14:56:25.036998 1197 catalog_manager.cc:4841] TS c9fce3001c8a4244b058751cace83dd0 (127.31.40.1:36267): tablet aa4581f30e4c432e914472b4cc005c4e (table test-table [id=ba30c56636bb4db19ea87ba1f747cd6d]) successfully deleted
Sep 28 14:56:25 dist-test-slave-j968 krb5kdc[1165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607385, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 14:56:25 dist-test-slave-j968 krb5kdc[1165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607385, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:56:25.078783 1823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:25.062341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43860 (local address 127.31.40.62:43811)
0928 14:56:25.062928 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:25.062932 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:25.062955 (+ 23us) server_negotiation.cc:407] Connection header received
0928 14:56:25.063133 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:25.063137 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:25.063199 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:25.063299 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:25.064899 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:25.065681 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:25.067093 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:25.067483 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:25.067585 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:25.071702 (+ 4117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:25.071707 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:25.071711 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:25.071748 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:25.075970 (+ 4222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:25.076734 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:25.076738 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:25.076741 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:25.076927 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:25.077270 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:25.077274 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:25.077276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:25.077673 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:25.077916 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:25.078151 (+ 235us) server_negotiation.cc:299] Negotiation successful
0928 14:56:25.078380 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":329,"threads_started":1}
I20260928 14:56:25.082779 1209 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43860:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:56:25.083561 1209 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:56:25.091785 1350 tablet_service.cc:1467] Processing CreateTablet for tablet 124c809c7e7e4b238099578fa4cf7171 (DEFAULT_TABLE table=test-table [id=ba3c6421d647483589b36f66b509abf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:56:25.092253 1491 tablet_service.cc:1467] Processing CreateTablet for tablet 124c809c7e7e4b238099578fa4cf7171 (DEFAULT_TABLE table=test-table [id=ba3c6421d647483589b36f66b509abf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:56:25.092399 1350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 124c809c7e7e4b238099578fa4cf7171. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:56:25.092284 1630 tablet_service.cc:1467] Processing CreateTablet for tablet 124c809c7e7e4b238099578fa4cf7171 (DEFAULT_TABLE table=test-table [id=ba3c6421d647483589b36f66b509abf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:56:25.092789 1491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 124c809c7e7e4b238099578fa4cf7171. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:56:25.092869 1630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 124c809c7e7e4b238099578fa4cf7171. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:56:25.101481 1824 tablet_bootstrap.cc:492] T 124c809c7e7e4b238099578fa4cf7171 P a3466cf67681447890434b1fc93bba03: Bootstrap starting.
I20260928 14:56:25.102250 1825 tablet_bootstrap.cc:492] T 124c809c7e7e4b238099578fa4cf7171 P c9fce3001c8a4244b058751cace83dd0: Bootstrap starting.
I20260928 14:56:25.104344 1825 tablet_bootstrap.cc:654] T 124c809c7e7e4b238099578fa4cf7171 P c9fce3001c8a4244b058751cace83dd0: Neither blocks nor log segments found. Creating new log.
I20260928 14:56:25.104878 1824 tablet_bootstrap.cc:654] T 124c809c7e7e4b238099578fa4cf7171 P a3466cf67681447890434b1fc93bba03: Neither blocks nor log segments found. Creating new log.
I20260928 14:56:25.106222 1826 tablet_bootstrap.cc:492] T 124c809c7e7e4b238099578fa4cf7171 P 85d458ba5c774fca927a3fcd6362fb71: Bootstrap starting.
I20260928 14:56:25.107064 1825 tablet_bootstrap.cc:492] T 124c809c7e7e4b238099578fa4cf7171 P c9fce3001c8a4244b058751cace83dd0: No bootstrap required, opened a new log
I20260928 14:56:25.107231 1825 ts_tablet_manager.cc:1397] T 124c809c7e7e4b238099578fa4cf7171 P c9fce3001c8a4244b058751cace83dd0: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 14:56:25.107692 1824 tablet_bootstrap.cc:492] T 124c809c7e7e4b238099578fa4cf7171 P a3466cf67681447890434b1fc93bba03: No bootstrap required, opened a new log
I20260928 14:56:25.107849 1824 ts_tablet_manager.cc:1397] T 124c809c7e7e4b238099578fa4cf7171 P a3466cf67681447890434b1fc93bba03: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260928 14:56:25.108147 1826 tablet_bootstrap.cc:654] T 124c809c7e7e4b238099578fa4cf7171 P 85d458ba5c774fca927a3fcd6362fb71: Neither blocks nor log segments found. Creating new log.
I20260928 14:56:25.108140 1825 raft_consensus.cc:348] T 124c809c7e7e4b238099578fa4cf7171 P c9fce3001c8a4244b058751cace83dd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 } } peers { permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 } } peers { permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 } }
I20260928 14:56:25.108390 1825 raft_consensus.cc:374] T 124c809c7e7e4b238099578fa4cf7171 P c9fce3001c8a4244b058751cace83dd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:56:25.108505 1825 raft_consensus.cc:729] T 124c809c7e7e4b238099578fa4cf7171 P c9fce3001c8a4244b058751cace83dd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9fce3001c8a4244b058751cace83dd0, State: Initialized, Role: FOLLOWER
I20260928 14:56:25.108737 1825 consensus_queue.cc:260] T 124c809c7e7e4b238099578fa4cf7171 P c9fce3001c8a4244b058751cace83dd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 } } peers { permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 } } peers { permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 } }
I20260928 14:56:25.109292 1825 ts_tablet_manager.cc:1428] T 124c809c7e7e4b238099578fa4cf7171 P c9fce3001c8a4244b058751cace83dd0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:56:25.110267 1824 raft_consensus.cc:348] T 124c809c7e7e4b238099578fa4cf7171 P a3466cf67681447890434b1fc93bba03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 } } peers { permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 } } peers { permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 } }
I20260928 14:56:25.110539 1824 raft_consensus.cc:374] T 124c809c7e7e4b238099578fa4cf7171 P a3466cf67681447890434b1fc93bba03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:56:25.110646 1824 raft_consensus.cc:729] T 124c809c7e7e4b238099578fa4cf7171 P a3466cf67681447890434b1fc93bba03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3466cf67681447890434b1fc93bba03, State: Initialized, Role: FOLLOWER
I20260928 14:56:25.111157 1826 tablet_bootstrap.cc:492] T 124c809c7e7e4b238099578fa4cf7171 P 85d458ba5c774fca927a3fcd6362fb71: No bootstrap required, opened a new log
I20260928 14:56:25.111083 1824 consensus_queue.cc:260] T 124c809c7e7e4b238099578fa4cf7171 P a3466cf67681447890434b1fc93bba03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 } } peers { permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 } } peers { permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 } }
I20260928 14:56:25.111509 1826 ts_tablet_manager.cc:1397] T 124c809c7e7e4b238099578fa4cf7171 P 85d458ba5c774fca927a3fcd6362fb71: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 14:56:25.112365 1824 ts_tablet_manager.cc:1428] T 124c809c7e7e4b238099578fa4cf7171 P a3466cf67681447890434b1fc93bba03: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260928 14:56:25.112406 1826 raft_consensus.cc:348] T 124c809c7e7e4b238099578fa4cf7171 P 85d458ba5c774fca927a3fcd6362fb71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 } } peers { permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 } } peers { permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 } }
I20260928 14:56:25.112695 1826 raft_consensus.cc:374] T 124c809c7e7e4b238099578fa4cf7171 P 85d458ba5c774fca927a3fcd6362fb71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:56:25.112824 1826 raft_consensus.cc:729] T 124c809c7e7e4b238099578fa4cf7171 P 85d458ba5c774fca927a3fcd6362fb71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85d458ba5c774fca927a3fcd6362fb71, State: Initialized, Role: FOLLOWER
I20260928 14:56:25.113091 1826 consensus_queue.cc:260] T 124c809c7e7e4b238099578fa4cf7171 P 85d458ba5c774fca927a3fcd6362fb71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 } } peers { permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 } } peers { permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 } }
I20260928 14:56:25.113999 1826 ts_tablet_manager.cc:1428] T 124c809c7e7e4b238099578fa4cf7171 P 85d458ba5c774fca927a3fcd6362fb71: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:56:25.238870 1726 raft_consensus.cc:482] T 124c809c7e7e4b238099578fa4cf7171 P c9fce3001c8a4244b058751cace83dd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:56:25.239084 1726 raft_consensus.cc:504] T 124c809c7e7e4b238099578fa4cf7171 P c9fce3001c8a4244b058751cace83dd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 } } peers { permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 } } peers { permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 } }
I20260928 14:56:25.239830 1726 leader_election.cc:288] T 124c809c7e7e4b238099578fa4cf7171 P c9fce3001c8a4244b058751cace83dd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85d458ba5c774fca927a3fcd6362fb71 (127.31.40.2:39879), a3466cf67681447890434b1fc93bba03 (127.31.40.3:33391)
I20260928 14:56:25.240483 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "124c809c7e7e4b238099578fa4cf7171" candidate_uuid: "c9fce3001c8a4244b058751cace83dd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3466cf67681447890434b1fc93bba03" is_pre_election: true
I20260928 14:56:25.240499 1511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "124c809c7e7e4b238099578fa4cf7171" candidate_uuid: "c9fce3001c8a4244b058751cace83dd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85d458ba5c774fca927a3fcd6362fb71" is_pre_election: true
I20260928 14:56:25.240717 1656 raft_consensus.cc:2454] T 124c809c7e7e4b238099578fa4cf7171 P a3466cf67681447890434b1fc93bba03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9fce3001c8a4244b058751cace83dd0 in term 0.
I20260928 14:56:25.240783 1511 raft_consensus.cc:2454] T 124c809c7e7e4b238099578fa4cf7171 P 85d458ba5c774fca927a3fcd6362fb71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9fce3001c8a4244b058751cace83dd0 in term 0.
I20260928 14:56:25.241257 1301 leader_election.cc:302] T 124c809c7e7e4b238099578fa4cf7171 P c9fce3001c8a4244b058751cace83dd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3466cf67681447890434b1fc93bba03, c9fce3001c8a4244b058751cace83dd0; no voters:
I20260928 14:56:25.241559 1726 raft_consensus.cc:2781] T 124c809c7e7e4b238099578fa4cf7171 P c9fce3001c8a4244b058751cace83dd0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:56:25.241669 1726 raft_consensus.cc:482] T 124c809c7e7e4b238099578fa4cf7171 P c9fce3001c8a4244b058751cace83dd0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:56:25.241739 1726 raft_consensus.cc:3037] T 124c809c7e7e4b238099578fa4cf7171 P c9fce3001c8a4244b058751cace83dd0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:56:25.244210 1726 raft_consensus.cc:504] T 124c809c7e7e4b238099578fa4cf7171 P c9fce3001c8a4244b058751cace83dd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 } } peers { permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 } } peers { permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 } }
I20260928 14:56:25.244990 1726 leader_election.cc:288] T 124c809c7e7e4b238099578fa4cf7171 P c9fce3001c8a4244b058751cace83dd0 [CANDIDATE]: Term 1 election: Requested vote from peers 85d458ba5c774fca927a3fcd6362fb71 (127.31.40.2:39879), a3466cf67681447890434b1fc93bba03 (127.31.40.3:33391)
I20260928 14:56:25.245533 1511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "124c809c7e7e4b238099578fa4cf7171" candidate_uuid: "c9fce3001c8a4244b058751cace83dd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85d458ba5c774fca927a3fcd6362fb71"
I20260928 14:56:25.245553 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "124c809c7e7e4b238099578fa4cf7171" candidate_uuid: "c9fce3001c8a4244b058751cace83dd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3466cf67681447890434b1fc93bba03"
I20260928 14:56:25.245814 1656 raft_consensus.cc:3037] T 124c809c7e7e4b238099578fa4cf7171 P a3466cf67681447890434b1fc93bba03 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:56:25.245813 1511 raft_consensus.cc:3037] T 124c809c7e7e4b238099578fa4cf7171 P 85d458ba5c774fca927a3fcd6362fb71 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:56:25.249178 1511 raft_consensus.cc:2454] T 124c809c7e7e4b238099578fa4cf7171 P 85d458ba5c774fca927a3fcd6362fb71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9fce3001c8a4244b058751cace83dd0 in term 1.
I20260928 14:56:25.249193 1656 raft_consensus.cc:2454] T 124c809c7e7e4b238099578fa4cf7171 P a3466cf67681447890434b1fc93bba03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9fce3001c8a4244b058751cace83dd0 in term 1.
I20260928 14:56:25.249735 1300 leader_election.cc:302] T 124c809c7e7e4b238099578fa4cf7171 P c9fce3001c8a4244b058751cace83dd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 85d458ba5c774fca927a3fcd6362fb71, a3466cf67681447890434b1fc93bba03, c9fce3001c8a4244b058751cace83dd0; no voters:
I20260928 14:56:25.250172 1726 raft_consensus.cc:2781] T 124c809c7e7e4b238099578fa4cf7171 P c9fce3001c8a4244b058751cace83dd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:56:25.250350 1726 raft_consensus.cc:686] T 124c809c7e7e4b238099578fa4cf7171 P c9fce3001c8a4244b058751cace83dd0 [term 1 LEADER]: Becoming Leader. State: Replica: c9fce3001c8a4244b058751cace83dd0, State: Running, Role: LEADER
I20260928 14:56:25.250666 1726 consensus_queue.cc:237] T 124c809c7e7e4b238099578fa4cf7171 P c9fce3001c8a4244b058751cace83dd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 } } peers { permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 } } peers { permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 } }
I20260928 14:56:25.253571 1209 catalog_manager.cc:5495] T 124c809c7e7e4b238099578fa4cf7171 P c9fce3001c8a4244b058751cace83dd0 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9fce3001c8a4244b058751cace83dd0 (127.31.40.1). New cstate: current_term: 1 leader_uuid: "c9fce3001c8a4244b058751cace83dd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 } health_report { overall_health: UNKNOWN } } }
I20260928 14:56:25.313424 1834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:25.304644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48462 (local address 127.31.40.1:36267)
0928 14:56:25.305825 (+ 1181us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:25.305829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:25.305854 (+ 25us) server_negotiation.cc:407] Connection header received
0928 14:56:25.305977 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:25.305981 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:25.306043 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:25.306133 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:56:25.306615 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:25.308093 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:25.310632 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:25.311284 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:25.311392 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:25.312420 (+ 1028us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:56:25.312653 (+ 233us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:56:25.312732 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:25.312880 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 14:56:25.312947 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1054,"thread_start_us":637,"threads_started":1}
I20260928 14:56:25.333191 1837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:25.324688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:36267 (local address 127.31.40.1:36329)
0928 14:56:25.326090 (+ 1402us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:25.326117 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:25.326277 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:25.326596 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:25.326604 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:25.326621 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:56:25.326937 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:25.326951 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:25.328484 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:25.328492 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:25.330651 (+ 2159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:25.330672 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:25.331662 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:25.331666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:25.331823 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:25.331909 (+ 86us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:56:25.332472 (+ 563us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:56:25.332481 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 14:56:25.332548 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 14:56:25.332747 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1205,"mutex_wait_us":37,"thread_start_us":934,"threads_started":1}
I20260928 14:56:25.333214 1834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:25.324890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:36329 (local address 127.31.40.1:36267)
0928 14:56:25.325135 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:25.325151 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:25.326236 (+ 1085us) server_negotiation.cc:407] Connection header received
0928 14:56:25.326405 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:25.326409 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:25.326474 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:25.326555 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:56:25.327198 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:25.328338 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:25.330834 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:25.331475 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:25.331582 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:25.332154 (+ 572us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:56:25.332353 (+ 199us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:56:25.332458 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:25.332589 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 14:56:25.332702 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":67}
I20260928 14:56:25.342525 1656 raft_consensus.cc:1261] T 124c809c7e7e4b238099578fa4cf7171 P a3466cf67681447890434b1fc93bba03 [term 1 FOLLOWER]: Refusing update from remote peer c9fce3001c8a4244b058751cace83dd0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:56:25.342767 1511 raft_consensus.cc:1261] T 124c809c7e7e4b238099578fa4cf7171 P 85d458ba5c774fca927a3fcd6362fb71 [term 1 FOLLOWER]: Refusing update from remote peer c9fce3001c8a4244b058751cace83dd0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:56:25.343179 1794 consensus_queue.cc:1035] T 124c809c7e7e4b238099578fa4cf7171 P c9fce3001c8a4244b058751cace83dd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:56:25.343747 1832 consensus_queue.cc:1035] T 124c809c7e7e4b238099578fa4cf7171 P c9fce3001c8a4244b058751cace83dd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:56:25.407342 1209 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:56:25.407636 1209 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:56:25.411429 1209 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b9e26c8ffca249b9a82a4ec88f68f23b: Sending DeleteTablet for 3 replicas of tablet 124c809c7e7e4b238099578fa4cf7171
I20260928 14:56:25.412364 1350 tablet_service.cc:1514] Processing DeleteTablet for tablet 124c809c7e7e4b238099578fa4cf7171 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:56:25 UTC) from {username='kudu', principal='kudu/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:48438
I20260928 14:56:25.413117 1491 tablet_service.cc:1514] Processing DeleteTablet for tablet 124c809c7e7e4b238099578fa4cf7171 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:56:25 UTC) from {username='kudu', principal='kudu/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:43434
I20260928 14:56:25.413187 1813 tablet_replica.cc:330] T 124c809c7e7e4b238099578fa4cf7171 P c9fce3001c8a4244b058751cace83dd0: stopping tablet replica
I20260928 14:56:25.413401 1814 tablet_replica.cc:330] T 124c809c7e7e4b238099578fa4cf7171 P 85d458ba5c774fca927a3fcd6362fb71: stopping tablet replica
I20260928 14:56:25.413471 1813 raft_consensus.cc:2229] T 124c809c7e7e4b238099578fa4cf7171 P c9fce3001c8a4244b058751cace83dd0 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:56:25.413579 1814 raft_consensus.cc:2229] T 124c809c7e7e4b238099578fa4cf7171 P 85d458ba5c774fca927a3fcd6362fb71 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:56:25.413782 1814 raft_consensus.cc:2258] T 124c809c7e7e4b238099578fa4cf7171 P 85d458ba5c774fca927a3fcd6362fb71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:56:25.414117 1813 raft_consensus.cc:2258] T 124c809c7e7e4b238099578fa4cf7171 P c9fce3001c8a4244b058751cace83dd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:56:25.414609 1630 tablet_service.cc:1514] Processing DeleteTablet for tablet 124c809c7e7e4b238099578fa4cf7171 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:56:25 UTC) from {username='kudu', principal='kudu/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:49986
I20260928 14:56:25.414952 1815 tablet_replica.cc:330] T 124c809c7e7e4b238099578fa4cf7171 P a3466cf67681447890434b1fc93bba03: stopping tablet replica
I20260928 14:56:25.415189 1815 raft_consensus.cc:2229] T 124c809c7e7e4b238099578fa4cf7171 P a3466cf67681447890434b1fc93bba03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:56:25.415273 1813 ts_tablet_manager.cc:1905] T 124c809c7e7e4b238099578fa4cf7171 P c9fce3001c8a4244b058751cace83dd0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:56:25.415423 1815 raft_consensus.cc:2258] T 124c809c7e7e4b238099578fa4cf7171 P a3466cf67681447890434b1fc93bba03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:56:25.416234 1209 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43860:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 14:56:25.416586 1814 ts_tablet_manager.cc:1905] T 124c809c7e7e4b238099578fa4cf7171 P 85d458ba5c774fca927a3fcd6362fb71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:56:25.416771 1815 ts_tablet_manager.cc:1905] T 124c809c7e7e4b238099578fa4cf7171 P a3466cf67681447890434b1fc93bba03: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 14:56:25.417160 1209 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:56:25.422235 1813 ts_tablet_manager.cc:1918] T 124c809c7e7e4b238099578fa4cf7171 P c9fce3001c8a4244b058751cace83dd0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:56:25.422394 1813 log.cc:1196] T 124c809c7e7e4b238099578fa4cf7171 P c9fce3001c8a4244b058751cace83dd0: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-0/wal/wals/124c809c7e7e4b238099578fa4cf7171
I20260928 14:56:25.422781 1813 ts_tablet_manager.cc:1939] T 124c809c7e7e4b238099578fa4cf7171 P c9fce3001c8a4244b058751cace83dd0: Deleting consensus metadata
I20260928 14:56:25.423494 1814 ts_tablet_manager.cc:1918] T 124c809c7e7e4b238099578fa4cf7171 P 85d458ba5c774fca927a3fcd6362fb71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:56:25.423626 1815 ts_tablet_manager.cc:1918] T 124c809c7e7e4b238099578fa4cf7171 P a3466cf67681447890434b1fc93bba03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:56:25.423669 1814 log.cc:1196] T 124c809c7e7e4b238099578fa4cf7171 P 85d458ba5c774fca927a3fcd6362fb71: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-1/wal/wals/124c809c7e7e4b238099578fa4cf7171
I20260928 14:56:25.423771 1815 log.cc:1196] T 124c809c7e7e4b238099578fa4cf7171 P a3466cf67681447890434b1fc93bba03: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-2/wal/wals/124c809c7e7e4b238099578fa4cf7171
I20260928 14:56:25.424121 1814 ts_tablet_manager.cc:1939] T 124c809c7e7e4b238099578fa4cf7171 P 85d458ba5c774fca927a3fcd6362fb71: Deleting consensus metadata
I20260928 14:56:25.424258 1815 ts_tablet_manager.cc:1939] T 124c809c7e7e4b238099578fa4cf7171 P a3466cf67681447890434b1fc93bba03: Deleting consensus metadata
I20260928 14:56:25.424575 1197 catalog_manager.cc:4841] TS c9fce3001c8a4244b058751cace83dd0 (127.31.40.1:36267): tablet 124c809c7e7e4b238099578fa4cf7171 (table test-table [id=ba3c6421d647483589b36f66b509abf8]) successfully deleted
I20260928 14:56:25.425226 1195 catalog_manager.cc:4841] TS 85d458ba5c774fca927a3fcd6362fb71 (127.31.40.2:39879): tablet 124c809c7e7e4b238099578fa4cf7171 (table test-table [id=ba3c6421d647483589b36f66b509abf8]) successfully deleted
I20260928 14:56:25.425259 1196 catalog_manager.cc:4841] TS a3466cf67681447890434b1fc93bba03 (127.31.40.3:33391): tablet 124c809c7e7e4b238099578fa4cf7171 (table test-table [id=ba3c6421d647483589b36f66b509abf8]) successfully deleted
I20260928 14:56:25.426064 1491 tablet_service.cc:1467] Processing CreateTablet for tablet 0570ea37a210479db6bc7bdac99dbef6 (DEFAULT_TABLE table=test-table [id=7146b9a2ec424d08be2d6aa6c8e6ed35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:56:25.426267 1630 tablet_service.cc:1467] Processing CreateTablet for tablet 0570ea37a210479db6bc7bdac99dbef6 (DEFAULT_TABLE table=test-table [id=7146b9a2ec424d08be2d6aa6c8e6ed35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:56:25.426476 1491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0570ea37a210479db6bc7bdac99dbef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:56:25.426453 1350 tablet_service.cc:1467] Processing CreateTablet for tablet 0570ea37a210479db6bc7bdac99dbef6 (DEFAULT_TABLE table=test-table [id=7146b9a2ec424d08be2d6aa6c8e6ed35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:56:25.426656 1630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0570ea37a210479db6bc7bdac99dbef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:56:25.426915 1350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0570ea37a210479db6bc7bdac99dbef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:56:25.433869 1826 tablet_bootstrap.cc:492] T 0570ea37a210479db6bc7bdac99dbef6 P 85d458ba5c774fca927a3fcd6362fb71: Bootstrap starting.
I20260928 14:56:25.435911 1825 tablet_bootstrap.cc:492] T 0570ea37a210479db6bc7bdac99dbef6 P c9fce3001c8a4244b058751cace83dd0: Bootstrap starting.
I20260928 14:56:25.435858 1826 tablet_bootstrap.cc:654] T 0570ea37a210479db6bc7bdac99dbef6 P 85d458ba5c774fca927a3fcd6362fb71: Neither blocks nor log segments found. Creating new log.
I20260928 14:56:25.437717 1825 tablet_bootstrap.cc:654] T 0570ea37a210479db6bc7bdac99dbef6 P c9fce3001c8a4244b058751cace83dd0: Neither blocks nor log segments found. Creating new log.
I20260928 14:56:25.438218 1824 tablet_bootstrap.cc:492] T 0570ea37a210479db6bc7bdac99dbef6 P a3466cf67681447890434b1fc93bba03: Bootstrap starting.
I20260928 14:56:25.441849 1824 tablet_bootstrap.cc:654] T 0570ea37a210479db6bc7bdac99dbef6 P a3466cf67681447890434b1fc93bba03: Neither blocks nor log segments found. Creating new log.
I20260928 14:56:25.452566 1824 tablet_bootstrap.cc:492] T 0570ea37a210479db6bc7bdac99dbef6 P a3466cf67681447890434b1fc93bba03: No bootstrap required, opened a new log
I20260928 14:56:25.452795 1824 ts_tablet_manager.cc:1397] T 0570ea37a210479db6bc7bdac99dbef6 P a3466cf67681447890434b1fc93bba03: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.006s
I20260928 14:56:25.453246 1826 tablet_bootstrap.cc:492] T 0570ea37a210479db6bc7bdac99dbef6 P 85d458ba5c774fca927a3fcd6362fb71: No bootstrap required, opened a new log
I20260928 14:56:25.453555 1826 ts_tablet_manager.cc:1397] T 0570ea37a210479db6bc7bdac99dbef6 P 85d458ba5c774fca927a3fcd6362fb71: Time spent bootstrapping tablet: real 0.020s user 0.014s sys 0.000s
I20260928 14:56:25.453890 1824 raft_consensus.cc:348] T 0570ea37a210479db6bc7bdac99dbef6 P a3466cf67681447890434b1fc93bba03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 } } peers { permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 } } peers { permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 } }
I20260928 14:56:25.454154 1824 raft_consensus.cc:374] T 0570ea37a210479db6bc7bdac99dbef6 P a3466cf67681447890434b1fc93bba03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:56:25.454249 1824 raft_consensus.cc:729] T 0570ea37a210479db6bc7bdac99dbef6 P a3466cf67681447890434b1fc93bba03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3466cf67681447890434b1fc93bba03, State: Initialized, Role: FOLLOWER
I20260928 14:56:25.454447 1825 tablet_bootstrap.cc:492] T 0570ea37a210479db6bc7bdac99dbef6 P c9fce3001c8a4244b058751cace83dd0: No bootstrap required, opened a new log
I20260928 14:56:25.454494 1826 raft_consensus.cc:348] T 0570ea37a210479db6bc7bdac99dbef6 P 85d458ba5c774fca927a3fcd6362fb71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 } } peers { permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 } } peers { permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 } }
I20260928 14:56:25.454560 1824 consensus_queue.cc:260] T 0570ea37a210479db6bc7bdac99dbef6 P a3466cf67681447890434b1fc93bba03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 } } peers { permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 } } peers { permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 } }
I20260928 14:56:25.454783 1825 ts_tablet_manager.cc:1397] T 0570ea37a210479db6bc7bdac99dbef6 P c9fce3001c8a4244b058751cace83dd0: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.004s
I20260928 14:56:25.454850 1826 raft_consensus.cc:374] T 0570ea37a210479db6bc7bdac99dbef6 P 85d458ba5c774fca927a3fcd6362fb71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:56:25.455026 1826 raft_consensus.cc:729] T 0570ea37a210479db6bc7bdac99dbef6 P 85d458ba5c774fca927a3fcd6362fb71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85d458ba5c774fca927a3fcd6362fb71, State: Initialized, Role: FOLLOWER
I20260928 14:56:25.455456 1826 consensus_queue.cc:260] T 0570ea37a210479db6bc7bdac99dbef6 P 85d458ba5c774fca927a3fcd6362fb71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 } } peers { permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 } } peers { permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 } }
I20260928 14:56:25.456204 1826 ts_tablet_manager.cc:1428] T 0570ea37a210479db6bc7bdac99dbef6 P 85d458ba5c774fca927a3fcd6362fb71: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 14:56:25.456591 1824 ts_tablet_manager.cc:1428] T 0570ea37a210479db6bc7bdac99dbef6 P a3466cf67681447890434b1fc93bba03: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260928 14:56:25.457317 1825 raft_consensus.cc:348] T 0570ea37a210479db6bc7bdac99dbef6 P c9fce3001c8a4244b058751cace83dd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 } } peers { permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 } } peers { permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 } }
I20260928 14:56:25.457695 1825 raft_consensus.cc:374] T 0570ea37a210479db6bc7bdac99dbef6 P c9fce3001c8a4244b058751cace83dd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:56:25.457813 1825 raft_consensus.cc:729] T 0570ea37a210479db6bc7bdac99dbef6 P c9fce3001c8a4244b058751cace83dd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9fce3001c8a4244b058751cace83dd0, State: Initialized, Role: FOLLOWER
I20260928 14:56:25.458096 1825 consensus_queue.cc:260] T 0570ea37a210479db6bc7bdac99dbef6 P c9fce3001c8a4244b058751cace83dd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 } } peers { permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 } } peers { permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 } }
I20260928 14:56:25.458676 1825 ts_tablet_manager.cc:1428] T 0570ea37a210479db6bc7bdac99dbef6 P c9fce3001c8a4244b058751cace83dd0: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260928 14:56:25.484694 1830 raft_consensus.cc:482] T 0570ea37a210479db6bc7bdac99dbef6 P 85d458ba5c774fca927a3fcd6362fb71 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:56:25.484977 1830 raft_consensus.cc:504] T 0570ea37a210479db6bc7bdac99dbef6 P 85d458ba5c774fca927a3fcd6362fb71 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 } } peers { permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 } } peers { permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 } }
I20260928 14:56:25.485968 1830 leader_election.cc:288] T 0570ea37a210479db6bc7bdac99dbef6 P 85d458ba5c774fca927a3fcd6362fb71 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9fce3001c8a4244b058751cace83dd0 (127.31.40.1:36267), a3466cf67681447890434b1fc93bba03 (127.31.40.3:33391)
I20260928 14:56:25.486395 1371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0570ea37a210479db6bc7bdac99dbef6" candidate_uuid: "85d458ba5c774fca927a3fcd6362fb71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9fce3001c8a4244b058751cace83dd0" is_pre_election: true
I20260928 14:56:25.486528 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0570ea37a210479db6bc7bdac99dbef6" candidate_uuid: "85d458ba5c774fca927a3fcd6362fb71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3466cf67681447890434b1fc93bba03" is_pre_election: true
I20260928 14:56:25.486665 1371 raft_consensus.cc:2454] T 0570ea37a210479db6bc7bdac99dbef6 P c9fce3001c8a4244b058751cace83dd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85d458ba5c774fca927a3fcd6362fb71 in term 0.
I20260928 14:56:25.486848 1655 raft_consensus.cc:2454] T 0570ea37a210479db6bc7bdac99dbef6 P a3466cf67681447890434b1fc93bba03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85d458ba5c774fca927a3fcd6362fb71 in term 0.
I20260928 14:56:25.487246 1443 leader_election.cc:302] T 0570ea37a210479db6bc7bdac99dbef6 P 85d458ba5c774fca927a3fcd6362fb71 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85d458ba5c774fca927a3fcd6362fb71, c9fce3001c8a4244b058751cace83dd0; no voters:
I20260928 14:56:25.487527 1830 raft_consensus.cc:2781] T 0570ea37a210479db6bc7bdac99dbef6 P 85d458ba5c774fca927a3fcd6362fb71 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:56:25.487639 1830 raft_consensus.cc:482] T 0570ea37a210479db6bc7bdac99dbef6 P 85d458ba5c774fca927a3fcd6362fb71 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:56:25.487757 1830 raft_consensus.cc:3037] T 0570ea37a210479db6bc7bdac99dbef6 P 85d458ba5c774fca927a3fcd6362fb71 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:56:25.489815 1830 raft_consensus.cc:504] T 0570ea37a210479db6bc7bdac99dbef6 P 85d458ba5c774fca927a3fcd6362fb71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 } } peers { permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 } } peers { permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 } }
I20260928 14:56:25.490418 1830 leader_election.cc:288] T 0570ea37a210479db6bc7bdac99dbef6 P 85d458ba5c774fca927a3fcd6362fb71 [CANDIDATE]: Term 1 election: Requested vote from peers c9fce3001c8a4244b058751cace83dd0 (127.31.40.1:36267), a3466cf67681447890434b1fc93bba03 (127.31.40.3:33391)
I20260928 14:56:25.490857 1371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0570ea37a210479db6bc7bdac99dbef6" candidate_uuid: "85d458ba5c774fca927a3fcd6362fb71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9fce3001c8a4244b058751cace83dd0"
I20260928 14:56:25.491088 1371 raft_consensus.cc:3037] T 0570ea37a210479db6bc7bdac99dbef6 P c9fce3001c8a4244b058751cace83dd0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:56:25.491113 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0570ea37a210479db6bc7bdac99dbef6" candidate_uuid: "85d458ba5c774fca927a3fcd6362fb71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3466cf67681447890434b1fc93bba03"
I20260928 14:56:25.491433 1655 raft_consensus.cc:3037] T 0570ea37a210479db6bc7bdac99dbef6 P a3466cf67681447890434b1fc93bba03 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:56:25.493718 1371 raft_consensus.cc:2454] T 0570ea37a210479db6bc7bdac99dbef6 P c9fce3001c8a4244b058751cace83dd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85d458ba5c774fca927a3fcd6362fb71 in term 1.
I20260928 14:56:25.494171 1443 leader_election.cc:302] T 0570ea37a210479db6bc7bdac99dbef6 P 85d458ba5c774fca927a3fcd6362fb71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85d458ba5c774fca927a3fcd6362fb71, c9fce3001c8a4244b058751cace83dd0; no voters:
I20260928 14:56:25.494447 1830 raft_consensus.cc:2781] T 0570ea37a210479db6bc7bdac99dbef6 P 85d458ba5c774fca927a3fcd6362fb71 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:56:25.494575 1655 raft_consensus.cc:2454] T 0570ea37a210479db6bc7bdac99dbef6 P a3466cf67681447890434b1fc93bba03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85d458ba5c774fca927a3fcd6362fb71 in term 1.
I20260928 14:56:25.504999 1830 raft_consensus.cc:686] T 0570ea37a210479db6bc7bdac99dbef6 P 85d458ba5c774fca927a3fcd6362fb71 [term 1 LEADER]: Becoming Leader. State: Replica: 85d458ba5c774fca927a3fcd6362fb71, State: Running, Role: LEADER
I20260928 14:56:25.505352 1830 consensus_queue.cc:237] T 0570ea37a210479db6bc7bdac99dbef6 P 85d458ba5c774fca927a3fcd6362fb71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 } } peers { permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 } } peers { permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 } }
I20260928 14:56:25.508523 1208 catalog_manager.cc:5495] T 0570ea37a210479db6bc7bdac99dbef6 P 85d458ba5c774fca927a3fcd6362fb71 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85d458ba5c774fca927a3fcd6362fb71 (127.31.40.2). New cstate: current_term: 1 leader_uuid: "85d458ba5c774fca927a3fcd6362fb71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85d458ba5c774fca927a3fcd6362fb71" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 } health_report { overall_health: UNKNOWN } } }
I20260928 14:56:25.547626 1846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:25.539913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43456 (local address 127.31.40.2:39879)
0928 14:56:25.540984 (+ 1071us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:25.540989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:25.541025 (+ 36us) server_negotiation.cc:407] Connection header received
0928 14:56:25.541086 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:25.541089 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:25.541151 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:25.541236 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:56:25.541845 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:25.542960 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:25.545082 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:25.545905 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:25.546036 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:25.546686 (+ 650us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:56:25.546820 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:56:25.546960 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:25.547152 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 14:56:25.547224 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":891,"thread_start_us":648,"threads_started":1}
I20260928 14:56:25.558620 1655 raft_consensus.cc:1261] T 0570ea37a210479db6bc7bdac99dbef6 P a3466cf67681447890434b1fc93bba03 [term 1 FOLLOWER]: Refusing update from remote peer 85d458ba5c774fca927a3fcd6362fb71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:56:25.559202 1371 raft_consensus.cc:1261] T 0570ea37a210479db6bc7bdac99dbef6 P c9fce3001c8a4244b058751cace83dd0 [term 1 FOLLOWER]: Refusing update from remote peer 85d458ba5c774fca927a3fcd6362fb71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:56:25.559203 1830 consensus_queue.cc:1035] T 0570ea37a210479db6bc7bdac99dbef6 P 85d458ba5c774fca927a3fcd6362fb71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3466cf67681447890434b1fc93bba03" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:56:25.560099 1830 consensus_queue.cc:1035] T 0570ea37a210479db6bc7bdac99dbef6 P 85d458ba5c774fca927a3fcd6362fb71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9fce3001c8a4244b058751cace83dd0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 14:56:25.587954 1796 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 14:56:27.192339 1211 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:56:27.192736 1211 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:56:27.197659 1211 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b9e26c8ffca249b9a82a4ec88f68f23b: Sending DeleteTablet for 3 replicas of tablet 0570ea37a210479db6bc7bdac99dbef6
I20260928 14:56:27.198627 1491 tablet_service.cc:1514] Processing DeleteTablet for tablet 0570ea37a210479db6bc7bdac99dbef6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:56:27 UTC) from {username='kudu', principal='kudu/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:43434
I20260928 14:56:27.198655 1350 tablet_service.cc:1514] Processing DeleteTablet for tablet 0570ea37a210479db6bc7bdac99dbef6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:56:27 UTC) from {username='kudu', principal='kudu/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:48438
I20260928 14:56:27.198779 1630 tablet_service.cc:1514] Processing DeleteTablet for tablet 0570ea37a210479db6bc7bdac99dbef6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:56:27 UTC) from {username='kudu', principal='kudu/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:49986
I20260928 14:56:27.199597 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 1284
I20260928 14:56:27.201637 1874 tablet_replica.cc:330] T 0570ea37a210479db6bc7bdac99dbef6 P 85d458ba5c774fca927a3fcd6362fb71: stopping tablet replica
I20260928 14:56:27.201983 1874 raft_consensus.cc:2229] T 0570ea37a210479db6bc7bdac99dbef6 P 85d458ba5c774fca927a3fcd6362fb71 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:56:27.202515 1874 raft_consensus.cc:2258] T 0570ea37a210479db6bc7bdac99dbef6 P 85d458ba5c774fca927a3fcd6362fb71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:56:27.203927 1874 ts_tablet_manager.cc:1905] T 0570ea37a210479db6bc7bdac99dbef6 P 85d458ba5c774fca927a3fcd6362fb71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:56:27.207918 1873 tablet_replica.cc:330] T 0570ea37a210479db6bc7bdac99dbef6 P c9fce3001c8a4244b058751cace83dd0: stopping tablet replica
I20260928 14:56:27.208513 1875 tablet_replica.cc:330] T 0570ea37a210479db6bc7bdac99dbef6 P a3466cf67681447890434b1fc93bba03: stopping tablet replica
I20260928 14:56:27.208763 1875 raft_consensus.cc:2229] T 0570ea37a210479db6bc7bdac99dbef6 P a3466cf67681447890434b1fc93bba03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:56:27.209060 1873 raft_consensus.cc:2229] T 0570ea37a210479db6bc7bdac99dbef6 P c9fce3001c8a4244b058751cace83dd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:56:27.209391 1875 raft_consensus.cc:2258] T 0570ea37a210479db6bc7bdac99dbef6 P a3466cf67681447890434b1fc93bba03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:56:27.209661 1873 raft_consensus.cc:2258] T 0570ea37a210479db6bc7bdac99dbef6 P c9fce3001c8a4244b058751cace83dd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:56:27.210774 1875 ts_tablet_manager.cc:1905] T 0570ea37a210479db6bc7bdac99dbef6 P a3466cf67681447890434b1fc93bba03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:56:27.211541 1873 ts_tablet_manager.cc:1905] T 0570ea37a210479db6bc7bdac99dbef6 P c9fce3001c8a4244b058751cace83dd0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:56:27.213300 1874 ts_tablet_manager.cc:1918] T 0570ea37a210479db6bc7bdac99dbef6 P 85d458ba5c774fca927a3fcd6362fb71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 14:56:27.213482 1874 log.cc:1196] T 0570ea37a210479db6bc7bdac99dbef6 P 85d458ba5c774fca927a3fcd6362fb71: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-1/wal/wals/0570ea37a210479db6bc7bdac99dbef6
I20260928 14:56:27.214052 1874 ts_tablet_manager.cc:1939] T 0570ea37a210479db6bc7bdac99dbef6 P 85d458ba5c774fca927a3fcd6362fb71: Deleting consensus metadata
I20260928 14:56:27.215293 1195 catalog_manager.cc:4841] TS 85d458ba5c774fca927a3fcd6362fb71 (127.31.40.2:39879): tablet 0570ea37a210479db6bc7bdac99dbef6 (table test-table [id=7146b9a2ec424d08be2d6aa6c8e6ed35]) successfully deleted
I20260928 14:56:27.219472 1873 ts_tablet_manager.cc:1918] T 0570ea37a210479db6bc7bdac99dbef6 P c9fce3001c8a4244b058751cace83dd0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 14:56:27.219743 1873 log.cc:1196] T 0570ea37a210479db6bc7bdac99dbef6 P c9fce3001c8a4244b058751cace83dd0: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-0/wal/wals/0570ea37a210479db6bc7bdac99dbef6
I20260928 14:56:27.219846 1875 ts_tablet_manager.cc:1918] T 0570ea37a210479db6bc7bdac99dbef6 P a3466cf67681447890434b1fc93bba03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 14:56:27.220088 1875 log.cc:1196] T 0570ea37a210479db6bc7bdac99dbef6 P a3466cf67681447890434b1fc93bba03: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607365362981-31904-0/minicluster-data/ts-2/wal/wals/0570ea37a210479db6bc7bdac99dbef6
I20260928 14:56:27.220367 1873 ts_tablet_manager.cc:1939] T 0570ea37a210479db6bc7bdac99dbef6 P c9fce3001c8a4244b058751cace83dd0: Deleting consensus metadata
I20260928 14:56:27.220708 1875 ts_tablet_manager.cc:1939] T 0570ea37a210479db6bc7bdac99dbef6 P a3466cf67681447890434b1fc93bba03: Deleting consensus metadata
I20260928 14:56:27.221540 1197 catalog_manager.cc:4841] TS c9fce3001c8a4244b058751cace83dd0 (127.31.40.1:36267): tablet 0570ea37a210479db6bc7bdac99dbef6 (table test-table [id=7146b9a2ec424d08be2d6aa6c8e6ed35]) successfully deleted
I20260928 14:56:27.221824 1196 catalog_manager.cc:4841] TS a3466cf67681447890434b1fc93bba03 (127.31.40.3:33391): tablet 0570ea37a210479db6bc7bdac99dbef6 (table test-table [id=7146b9a2ec424d08be2d6aa6c8e6ed35]) successfully deleted
Sep 28 14:56:27 dist-test-slave-j968 krb5kdc[1165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607385, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.40.1@KRBTEST.COM
I20260928 14:56:27.241053 1877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:27.201557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48478 (local address 127.31.40.1:36267)
0928 14:56:27.215550 (+ 13993us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:27.215556 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:27.215581 (+ 25us) server_negotiation.cc:407] Connection header received
0928 14:56:27.215639 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:27.215644 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:27.215736 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:27.215859 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:27.217667 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:27.218958 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:27.221233 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:27.221957 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:27.222091 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:27.234667 (+ 12576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:27.234685 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:27.234702 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:27.234745 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:27.237813 (+ 3068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:27.238737 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:27.238745 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:27.238748 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:27.239023 (+ 275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:27.239464 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:27.239467 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:27.239475 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:27.239990 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:27.240247 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:27.240562 (+ 315us) server_negotiation.cc:299] Negotiation successful
0928 14:56:27.240752 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13824,"thread_start_us":13999,"threads_started":1}
W20260928 14:56:27.241828 1416 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48478
W20260928 14:56:27.242625 31904 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 1284: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 14:56:27.242897 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 1284
I20260928 14:56:27.269827 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 1425
Sep 28 14:56:27 dist-test-slave-j968 krb5kdc[1165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607385, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.40.2@KRBTEST.COM
I20260928 14:56:27.287102 1878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:27.270573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43460 (local address 127.31.40.2:39879)
0928 14:56:27.271117 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:27.271121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:27.271138 (+ 17us) server_negotiation.cc:407] Connection header received
0928 14:56:27.271195 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:27.271199 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:27.271255 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:27.271351 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:27.273079 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:27.273856 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:27.275202 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:27.275645 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:27.275748 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:27.280580 (+ 4832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:27.280598 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:27.280617 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:27.280664 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:27.284167 (+ 3503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:27.285045 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:27.285053 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:27.285056 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:27.285204 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:27.285585 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:27.285589 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:27.285591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:27.286002 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:27.286265 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:27.286577 (+ 312us) server_negotiation.cc:299] Negotiation successful
0928 14:56:27.286751 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":289,"threads_started":1}
W20260928 14:56:27.287947 1556 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43460
W20260928 14:56:27.288673 31904 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 1425: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 14:56:27.288861 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 1425
I20260928 14:56:27.312364 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 1565
Sep 28 14:56:27 dist-test-slave-j968 krb5kdc[1165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607385, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.40.3@KRBTEST.COM
I20260928 14:56:27.328863 1879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:27.313124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50002 (local address 127.31.40.3:33391)
0928 14:56:27.313644 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:27.313648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:27.313665 (+ 17us) server_negotiation.cc:407] Connection header received
0928 14:56:27.313720 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:27.313724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:27.313787 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:27.313878 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:27.315259 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:27.316261 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:27.317532 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:27.317906 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:27.318021 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:27.322689 (+ 4668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:27.322705 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:27.322722 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:27.322761 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:27.325771 (+ 3010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:27.326558 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:27.326566 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:27.326568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:27.326704 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:27.327279 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:27.327282 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:27.327284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:27.327661 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:27.328053 (+ 392us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:27.328385 (+ 332us) server_negotiation.cc:299] Negotiation successful
0928 14:56:27.328593 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":291,"threads_started":1}
W20260928 14:56:27.329665 1696 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50002
W20260928 14:56:27.330492 31904 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 1565: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 14:56:27.330700 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 1565
I20260928 14:56:27.353673 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 1179
I20260928 14:56:27.355041 1252 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:56:27.476075 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 1179
2026-09-28T14:56:27Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8556 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:56:27 dist-test-slave-j968 krb5kdc[1883](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:56:27 dist-test-slave-j968 krb5kdc[1883](info): set up 2 sockets
Sep 28 14:56:27 dist-test-slave-j968 krb5kdc[1883](info): commencing operation
krb5kdc: starting...
W20260928 14:56:28.602496 31904 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.001s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Sep 28 14:56:28 dist-test-slave-j968 krb5kdc[1883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:56:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:56:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab.
Entry for principal kudu/127.31.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab.
I20260928 14:56:28.820719 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:45209
--webserver_interface=127.31.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:36851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.40.62:45209
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:56:29.171854 1896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:56:29.172187 1896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:56:29.172386 1896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:56:29.179769 1896 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:56:29.179909 1896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:56:29.179960 1896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:56:29.180047 1896 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:56:29.180114 1896 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:56:29.190228 1896 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:36851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.40.62:45209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:45209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:56:29.191696 1896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:56:29.193143 1896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:56:29.206493 1901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:29.206813 1896 server_base.cc:1034] running on GCE node
W20260928 14:56:29.206436 1902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:29.206466 1904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:29.208248 1896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:56:29.209823 1896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:56:29.211109 1896 hybrid_clock.cc:648] HybridClock initialized: now 1790607389210990 us; error 58 us; skew 500 ppm
I20260928 14:56:29.211490 1896 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:56:29 dist-test-slave-j968 krb5kdc[1883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607389, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:56:29.214932 1896 init.cc:377] Logged in from keytab as kudu/127.31.40.62@KRBTEST.COM (short username kudu)
I20260928 14:56:29.216681 1896 webserver.cc:460] Webserver started at http://127.31.40.62:37995/ using document root <none> and password file <none>
I20260928 14:56:29.217440 1896 fs_manager.cc:359] Metadata directory not provided
I20260928 14:56:29.217592 1896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:56:29.217921 1896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:56:29.221073 1896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/master-0/data/instance:
uuid: "d827684651e14c5db77c6ed16ebffa9f"
format_stamp: "Formatted at 2026-09-28 14:56:29 on dist-test-slave-j968"
server_key: "6c2519c4c2fb1150cf139650b69bb4c6"
server_key_iv: "ec130e1189e66d622e77b153905ff1c5"
server_key_version: "encryptionkey@0"
I20260928 14:56:29.221851 1896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/master-0/wal/instance:
uuid: "d827684651e14c5db77c6ed16ebffa9f"
format_stamp: "Formatted at 2026-09-28 14:56:29 on dist-test-slave-j968"
server_key: "6c2519c4c2fb1150cf139650b69bb4c6"
server_key_iv: "ec130e1189e66d622e77b153905ff1c5"
server_key_version: "encryptionkey@0"
I20260928 14:56:29.228116 1896 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260928 14:56:29.232144 1911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:29.233578 1896 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:56:29.233779 1896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/master-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/master-0/wal
uuid: "d827684651e14c5db77c6ed16ebffa9f"
format_stamp: "Formatted at 2026-09-28 14:56:29 on dist-test-slave-j968"
server_key: "6c2519c4c2fb1150cf139650b69bb4c6"
server_key_iv: "ec130e1189e66d622e77b153905ff1c5"
server_key_version: "encryptionkey@0"
I20260928 14:56:29.233994 1896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:56:29.258385 1896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:56:29.262812 1896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:56:29.263114 1896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:56:29.282037 1896 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.62:45209
I20260928 14:56:29.282125 1963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.62:45209 every 8 connection(s)
I20260928 14:56:29.283423 1896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
I20260928 14:56:29.287614 1964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:56:29.293087 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 1896
I20260928 14:56:29.293347 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/master-0/wal/instance
I20260928 14:56:29.293965 31904 external_mini_cluster.cc:1428] Setting key 460f33eee8d13b7ae539bc7a9cb19eec
I20260928 14:56:29.302071 1964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d827684651e14c5db77c6ed16ebffa9f: Bootstrap starting.
I20260928 14:56:29.306314 1964 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d827684651e14c5db77c6ed16ebffa9f: Neither blocks nor log segments found. Creating new log.
I20260928 14:56:29.307811 1964 log.cc:824] T 00000000000000000000000000000000 P d827684651e14c5db77c6ed16ebffa9f: Log is configured to *not* fsync() on all Append() calls
Sep 28 14:56:29 dist-test-slave-j968 krb5kdc[1883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:56:29.311419 1964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d827684651e14c5db77c6ed16ebffa9f: No bootstrap required, opened a new log
I20260928 14:56:29.321201 1967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:29.296108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57464 (local address 127.31.40.62:45209)
0928 14:56:29.296950 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:29.296968 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:29.297025 (+ 57us) server_negotiation.cc:407] Connection header received
0928 14:56:29.300030 (+ 3005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:29.300064 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:29.300641 (+ 577us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:29.301148 (+ 507us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:29.302589 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:29.304957 (+ 2368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:29.306375 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:29.307574 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:29.307787 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:29.312743 (+ 4956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:29.312751 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:29.312767 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:29.312806 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:29.316391 (+ 3585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:29.317304 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:29.317325 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:29.317332 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:29.317540 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:29.318014 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:29.318019 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:29.318021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:29.318675 (+ 654us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:29.318894 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:29.319406 (+ 512us) server_negotiation.cc:299] Negotiation successful
0928 14:56:29.319926 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":337,"threads_started":1}
I20260928 14:56:29.321625 1964 raft_consensus.cc:348] T 00000000000000000000000000000000 P d827684651e14c5db77c6ed16ebffa9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d827684651e14c5db77c6ed16ebffa9f" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 45209 } }
I20260928 14:56:29.322113 1964 raft_consensus.cc:374] T 00000000000000000000000000000000 P d827684651e14c5db77c6ed16ebffa9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:56:29.322258 1964 raft_consensus.cc:729] T 00000000000000000000000000000000 P d827684651e14c5db77c6ed16ebffa9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d827684651e14c5db77c6ed16ebffa9f, State: Initialized, Role: FOLLOWER
I20260928 14:56:29.322862 1964 consensus_queue.cc:260] T 00000000000000000000000000000000 P d827684651e14c5db77c6ed16ebffa9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d827684651e14c5db77c6ed16ebffa9f" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 45209 } }
I20260928 14:56:29.323141 1964 raft_consensus.cc:388] T 00000000000000000000000000000000 P d827684651e14c5db77c6ed16ebffa9f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:56:29.323315 1964 raft_consensus.cc:482] T 00000000000000000000000000000000 P d827684651e14c5db77c6ed16ebffa9f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:56:29.323509 1964 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d827684651e14c5db77c6ed16ebffa9f [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:56:29.326062 1964 raft_consensus.cc:504] T 00000000000000000000000000000000 P d827684651e14c5db77c6ed16ebffa9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d827684651e14c5db77c6ed16ebffa9f" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 45209 } }
I20260928 14:56:29.326606 1964 leader_election.cc:302] T 00000000000000000000000000000000 P d827684651e14c5db77c6ed16ebffa9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d827684651e14c5db77c6ed16ebffa9f; no voters:
I20260928 14:56:29.327378 1964 leader_election.cc:288] T 00000000000000000000000000000000 P d827684651e14c5db77c6ed16ebffa9f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:56:29.327538 1969 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d827684651e14c5db77c6ed16ebffa9f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:56:29.328392 1969 raft_consensus.cc:686] T 00000000000000000000000000000000 P d827684651e14c5db77c6ed16ebffa9f [term 1 LEADER]: Becoming Leader. State: Replica: d827684651e14c5db77c6ed16ebffa9f, State: Running, Role: LEADER
I20260928 14:56:29.329052 1969 consensus_queue.cc:237] T 00000000000000000000000000000000 P d827684651e14c5db77c6ed16ebffa9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d827684651e14c5db77c6ed16ebffa9f" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 45209 } }
I20260928 14:56:29.329979 1964 sys_catalog.cc:564] T 00000000000000000000000000000000 P d827684651e14c5db77c6ed16ebffa9f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:56:29.332536 1970 sys_catalog.cc:455] T 00000000000000000000000000000000 P d827684651e14c5db77c6ed16ebffa9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d827684651e14c5db77c6ed16ebffa9f. Latest consensus state: current_term: 1 leader_uuid: "d827684651e14c5db77c6ed16ebffa9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d827684651e14c5db77c6ed16ebffa9f" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 45209 } } }
I20260928 14:56:29.332617 1971 sys_catalog.cc:455] T 00000000000000000000000000000000 P d827684651e14c5db77c6ed16ebffa9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d827684651e14c5db77c6ed16ebffa9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d827684651e14c5db77c6ed16ebffa9f" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 45209 } } }
I20260928 14:56:29.332957 1970 sys_catalog.cc:458] T 00000000000000000000000000000000 P d827684651e14c5db77c6ed16ebffa9f [sys.catalog]: This master's current role is: LEADER
I20260928 14:56:29.332984 1971 sys_catalog.cc:458] T 00000000000000000000000000000000 P d827684651e14c5db77c6ed16ebffa9f [sys.catalog]: This master's current role is: LEADER
I20260928 14:56:29.334857 1976 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:56:29.341611 1976 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:56:29.351492 1976 catalog_manager.cc:1347] Generated new cluster ID: a5dee6bf16d84005976b7d21dfdb52d2
I20260928 14:56:29.351644 1976 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:56:29.371052 1976 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:56:29.372555 1976 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:56:29.384974 1976 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d827684651e14c5db77c6ed16ebffa9f: Generated new TSK 0
I20260928 14:56:29.386008 1976 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.40.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab.
Entry for principal kudu/127.31.40.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab.
I20260928 14:56:29.448150 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.1:0
--local_ip_for_outbound_sockets=127.31.40.1
--webserver_interface=127.31.40.1
--webserver_port=0
--tserver_master_addrs=127.31.40.62:45209
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab
--principal=kudu/127.31.40.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:36851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:56:29.813715 1991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:56:29.814018 1991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:56:29.814177 1991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:56:29.821607 1991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:56:29.821782 1991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:56:29.822038 1991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.1
I20260928 14:56:29.832448 1991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:36851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab
--principal=kudu/127.31.40.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.40.1
--webserver_port=0
--tserver_master_addrs=127.31.40.62:45209
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.1
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:56:29.833936 1991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:56:29.835462 1991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:56:29.849156 1996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:29.849133 1997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:29.850410 1999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:29.851112 1991 server_base.cc:1034] running on GCE node
I20260928 14:56:29.851797 1991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:56:29.852967 1991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:56:29.854202 1991 hybrid_clock.cc:648] HybridClock initialized: now 1790607389854165 us; error 57 us; skew 500 ppm
I20260928 14:56:29.854543 1991 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:56:29 dist-test-slave-j968 krb5kdc[1883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607389, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:56:29.858201 1991 init.cc:377] Logged in from keytab as kudu/127.31.40.1@KRBTEST.COM (short username kudu)
I20260928 14:56:29.859743 1991 webserver.cc:460] Webserver started at http://127.31.40.1:36825/ using document root <none> and password file <none>
I20260928 14:56:29.860641 1991 fs_manager.cc:359] Metadata directory not provided
I20260928 14:56:29.860780 1991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:56:29.861140 1991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:56:29.864210 1991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-0/data/instance:
uuid: "c8d874a972834c729235aaa18b59a485"
format_stamp: "Formatted at 2026-09-28 14:56:29 on dist-test-slave-j968"
server_key: "0c2b2df25fc9ba1ff1aa0031cca87cf8"
server_key_iv: "51630e00998c17e27c40474823d106ed"
server_key_version: "encryptionkey@0"
I20260928 14:56:29.864974 1991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-0/wal/instance:
uuid: "c8d874a972834c729235aaa18b59a485"
format_stamp: "Formatted at 2026-09-28 14:56:29 on dist-test-slave-j968"
server_key: "0c2b2df25fc9ba1ff1aa0031cca87cf8"
server_key_iv: "51630e00998c17e27c40474823d106ed"
server_key_version: "encryptionkey@0"
I20260928 14:56:29.870396 1991 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260928 14:56:29.874313 2006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:29.876102 1991 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:56:29.876248 1991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-0/wal
uuid: "c8d874a972834c729235aaa18b59a485"
format_stamp: "Formatted at 2026-09-28 14:56:29 on dist-test-slave-j968"
server_key: "0c2b2df25fc9ba1ff1aa0031cca87cf8"
server_key_iv: "51630e00998c17e27c40474823d106ed"
server_key_version: "encryptionkey@0"
I20260928 14:56:29.876459 1991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:56:29.908742 1991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:56:29.914048 1991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:56:29.914382 1991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:56:29.915566 1991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:56:29.917116 1991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:56:29.917194 1991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:29.917284 1991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:56:29.917333 1991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:29.953580 1991 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.1:39679
I20260928 14:56:29.953672 2119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.1:39679 every 8 connection(s)
I20260928 14:56:29.954918 1991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
I20260928 14:56:29.962675 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 1991
I20260928 14:56:29.962984 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-0/wal/instance
I20260928 14:56:29.963843 31904 external_mini_cluster.cc:1428] Setting key 260107d875e39035db802a1be68256d2
Sep 28 14:56:29 dist-test-slave-j968 krb5kdc[1883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607389, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.1@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:56:29.977708 2122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:29.957829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:50397 (local address 127.31.40.62:45209)
0928 14:56:29.958304 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:29.958309 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:29.959366 (+ 1057us) server_negotiation.cc:407] Connection header received
0928 14:56:29.961313 (+ 1947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:29.961318 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:29.961429 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:29.961576 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:29.963279 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:29.964404 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:29.966304 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:29.967030 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:29.967198 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:29.970772 (+ 3574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:29.970782 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:29.970791 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:29.970856 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:29.973886 (+ 3030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:29.975037 (+ 1151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:29.975042 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:29.975044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:29.975169 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:29.975726 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:29.975730 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:29.975732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:29.976131 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:29.976255 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:29.977234 (+ 979us) server_negotiation.cc:299] Negotiation successful
0928 14:56:29.977457 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":235,"threads_started":1}
I20260928 14:56:29.979305 2123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:29.958420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:45209 (local address 127.31.40.1:50397)
0928 14:56:29.959187 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:29.959234 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:29.961028 (+ 1794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:29.961790 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:29.961799 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:29.962384 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:56:29.963057 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:29.963102 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:29.964600 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:29.964606 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:29.966135 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:29.966147 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:29.967271 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:29.967276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:29.967493 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:29.968258 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:56:29.968282 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:56:29.970518 (+ 2236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:56:29.974125 (+ 3607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:29.974137 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:29.974161 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:29.974842 (+ 681us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:29.975284 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:29.975293 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:29.975299 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:29.975573 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:29.976267 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:56:29.976277 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:56:29.976789 (+ 512us) client_negotiation.cc:769] Sending connection context
0928 14:56:29.977110 (+ 321us) client_negotiation.cc:240] Negotiation successful
0928 14:56:29.977729 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":302,"threads_started":1}
I20260928 14:56:29.981156 2120 heartbeater.cc:347] Connected to a master server at 127.31.40.62:45209
I20260928 14:56:29.981485 2120 heartbeater.cc:464] Registering TS with master...
I20260928 14:56:29.982270 2120 heartbeater.cc:511] Master 127.31.40.62:45209 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.31.40.2@KRBTEST.COM; defaulting to no policy
I20260928 14:56:29.984537 1929 ts_manager.cc:194] Registered new tserver with Master: c8d874a972834c729235aaa18b59a485 (127.31.40.1:39679)
I20260928 14:56:29.986179 1929 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.1@KRBTEST.COM'} at 127.31.40.1:50397
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab.
Entry for principal kudu/127.31.40.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab.
I20260928 14:56:30.012197 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.2:0
--local_ip_for_outbound_sockets=127.31.40.2
--webserver_interface=127.31.40.2
--webserver_port=0
--tserver_master_addrs=127.31.40.62:45209
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab
--principal=kudu/127.31.40.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:36851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:56:30.355342 2126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:56:30.355693 2126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:56:30.355909 2126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:56:30.364321 2126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:56:30.364487 2126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:56:30.364683 2126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.2
I20260928 14:56:30.374970 2126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:36851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab
--principal=kudu/127.31.40.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.40.2
--webserver_port=0
--tserver_master_addrs=127.31.40.62:45209
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.2
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:56:30.376380 2126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:56:30.377799 2126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:56:30.391994 2131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:30.391923 2132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:30.392475 2134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:30.393718 2126 server_base.cc:1034] running on GCE node
I20260928 14:56:30.394400 2126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:56:30.395368 2126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:56:30.396616 2126 hybrid_clock.cc:648] HybridClock initialized: now 1790607390396529 us; error 55 us; skew 500 ppm
I20260928 14:56:30.396960 2126 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:56:30 dist-test-slave-j968 krb5kdc[1883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607390, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:56:30.400332 2126 init.cc:377] Logged in from keytab as kudu/127.31.40.2@KRBTEST.COM (short username kudu)
I20260928 14:56:30.402119 2126 webserver.cc:460] Webserver started at http://127.31.40.2:41083/ using document root <none> and password file <none>
I20260928 14:56:30.402860 2126 fs_manager.cc:359] Metadata directory not provided
I20260928 14:56:30.402998 2126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:56:30.403319 2126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:56:30.406399 2126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-1/data/instance:
uuid: "9703d0c6d9c44315a23945a59549d990"
format_stamp: "Formatted at 2026-09-28 14:56:30 on dist-test-slave-j968"
server_key: "ed87fd9baf03570666f1e93005f479b7"
server_key_iv: "0c836c45e031b29b1e683dd2a9ac148d"
server_key_version: "encryptionkey@0"
I20260928 14:56:30.407208 2126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-1/wal/instance:
uuid: "9703d0c6d9c44315a23945a59549d990"
format_stamp: "Formatted at 2026-09-28 14:56:30 on dist-test-slave-j968"
server_key: "ed87fd9baf03570666f1e93005f479b7"
server_key_iv: "0c836c45e031b29b1e683dd2a9ac148d"
server_key_version: "encryptionkey@0"
I20260928 14:56:30.413105 2126 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260928 14:56:30.416734 2141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:30.418414 2126 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:56:30.418609 2126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-1/wal
uuid: "9703d0c6d9c44315a23945a59549d990"
format_stamp: "Formatted at 2026-09-28 14:56:30 on dist-test-slave-j968"
server_key: "ed87fd9baf03570666f1e93005f479b7"
server_key_iv: "0c836c45e031b29b1e683dd2a9ac148d"
server_key_version: "encryptionkey@0"
I20260928 14:56:30.418849 2126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:56:30.458899 2126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:56:30.465852 2126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:56:30.466203 2126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:56:30.467360 2126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:56:30.468962 2126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:56:30.469079 2126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:30.469249 2126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:56:30.469331 2126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:30.506832 2126 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.2:36285
I20260928 14:56:30.506882 2254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.2:36285 every 8 connection(s)
I20260928 14:56:30.508440 2126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
I20260928 14:56:30.513345 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 2126
I20260928 14:56:30.513564 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-1/wal/instance
I20260928 14:56:30.514025 31904 external_mini_cluster.cc:1428] Setting key c7add7b185297d2c4cdbc31a2fde539d
Sep 28 14:56:30 dist-test-slave-j968 krb5kdc[1883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607390, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.2@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:56:30.532423 2257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:30.511216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:59527 (local address 127.31.40.62:45209)
0928 14:56:30.511679 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:30.511683 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:30.512801 (+ 1118us) server_negotiation.cc:407] Connection header received
0928 14:56:30.514909 (+ 2108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:30.514916 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:30.515093 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:30.515214 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:30.517206 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:30.518529 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:30.520333 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:30.521134 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:30.521287 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:30.525680 (+ 4393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:30.525684 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:30.525688 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:30.525723 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:30.528488 (+ 2765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:30.529338 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:30.529347 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:30.529354 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:30.529527 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:30.530094 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:30.530101 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:30.530107 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:30.530537 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:30.530715 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:30.531677 (+ 962us) server_negotiation.cc:299] Negotiation successful
0928 14:56:30.531975 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":246,"threads_started":1}
I20260928 14:56:30.533269 2258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:30.511924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:45209 (local address 127.31.40.2:59527)
0928 14:56:30.512635 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:30.512680 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:30.514549 (+ 1869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:30.515524 (+ 975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:30.515537 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:30.516369 (+ 832us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:56:30.517004 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:30.517025 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:30.518707 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:30.518711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:30.520174 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:30.520189 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:30.521386 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:30.521391 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:30.521628 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:30.522430 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:56:30.522463 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:56:30.525419 (+ 2956us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:56:30.528645 (+ 3226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:30.528655 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:30.528681 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:30.529156 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:30.529683 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:30.529690 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:30.529694 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:30.529885 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:30.530694 (+ 809us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:56:30.530700 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:56:30.531211 (+ 511us) client_negotiation.cc:769] Sending connection context
0928 14:56:30.531535 (+ 324us) client_negotiation.cc:240] Negotiation successful
0928 14:56:30.532020 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":270,"threads_started":1}
WARNING: no policy specified for kudu/127.31.40.3@KRBTEST.COM; defaulting to no policy
I20260928 14:56:30.534869 2255 heartbeater.cc:347] Connected to a master server at 127.31.40.62:45209
I20260928 14:56:30.535235 2255 heartbeater.cc:464] Registering TS with master...
I20260928 14:56:30.536201 2255 heartbeater.cc:511] Master 127.31.40.62:45209 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.3@KRBTEST.COM" created.
I20260928 14:56:30.537949 1929 ts_manager.cc:194] Registered new tserver with Master: 9703d0c6d9c44315a23945a59549d990 (127.31.40.2:36285)
I20260928 14:56:30.539016 1929 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.2@KRBTEST.COM'} at 127.31.40.2:59527
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab.
Entry for principal kudu/127.31.40.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab.
I20260928 14:56:30.564831 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.3:0
--local_ip_for_outbound_sockets=127.31.40.3
--webserver_interface=127.31.40.3
--webserver_port=0
--tserver_master_addrs=127.31.40.62:45209
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab
--principal=kudu/127.31.40.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:36851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:56:30.904697 2261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:56:30.904992 2261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:56:30.905164 2261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:56:30.912528 2261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:56:30.912648 2261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:56:30.912840 2261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.3
I20260928 14:56:30.923074 2261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:36851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab
--principal=kudu/127.31.40.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.40.3
--webserver_port=0
--tserver_master_addrs=127.31.40.62:45209
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.3
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:56:30.924594 2261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:56:30.925945 2261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:56:30.940721 2269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:30.940747 2266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:30.940721 2267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:30.942219 2261 server_base.cc:1034] running on GCE node
I20260928 14:56:30.942879 2261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:56:30.944043 2261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:56:30.945282 2261 hybrid_clock.cc:648] HybridClock initialized: now 1790607390945232 us; error 45 us; skew 500 ppm
I20260928 14:56:30.945623 2261 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:56:30 dist-test-slave-j968 krb5kdc[1883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607390, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:56:30.948966 2261 init.cc:377] Logged in from keytab as kudu/127.31.40.3@KRBTEST.COM (short username kudu)
I20260928 14:56:30.950555 2261 webserver.cc:460] Webserver started at http://127.31.40.3:43327/ using document root <none> and password file <none>
I20260928 14:56:30.951220 2261 fs_manager.cc:359] Metadata directory not provided
I20260928 14:56:30.951315 2261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:56:30.951572 2261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:56:30.955335 2261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-2/data/instance:
uuid: "3cad188a79a3414fb7b1e2b879bd5aee"
format_stamp: "Formatted at 2026-09-28 14:56:30 on dist-test-slave-j968"
server_key: "8a19c42b38aca72dce2c1f88e95c71d1"
server_key_iv: "6fd5c4a4a1aed2fe2b1bf4f5a7126b44"
server_key_version: "encryptionkey@0"
I20260928 14:56:30.956395 2261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-2/wal/instance:
uuid: "3cad188a79a3414fb7b1e2b879bd5aee"
format_stamp: "Formatted at 2026-09-28 14:56:30 on dist-test-slave-j968"
server_key: "8a19c42b38aca72dce2c1f88e95c71d1"
server_key_iv: "6fd5c4a4a1aed2fe2b1bf4f5a7126b44"
server_key_version: "encryptionkey@0"
I20260928 14:56:30.962534 2261 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260928 14:56:30.966276 2276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:30.967778 2261 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:56:30.967993 2261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-2/wal
uuid: "3cad188a79a3414fb7b1e2b879bd5aee"
format_stamp: "Formatted at 2026-09-28 14:56:30 on dist-test-slave-j968"
server_key: "8a19c42b38aca72dce2c1f88e95c71d1"
server_key_iv: "6fd5c4a4a1aed2fe2b1bf4f5a7126b44"
server_key_version: "encryptionkey@0"
I20260928 14:56:30.968196 2261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:56:30.989594 2120 heartbeater.cc:503] Master 127.31.40.62:45209 was elected leader, sending a full tablet report...
I20260928 14:56:30.999646 2261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:56:31.004304 2261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:56:31.004624 2261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:56:31.005705 2261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:56:31.007133 2261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:56:31.007206 2261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:31.007333 2261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:56:31.007393 2261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:31.042083 2261 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.3:38451
I20260928 14:56:31.042210 2389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.3:38451 every 8 connection(s)
I20260928 14:56:31.043813 2261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
I20260928 14:56:31.046149 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 2261
I20260928 14:56:31.046362 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-2/wal/instance
I20260928 14:56:31.046808 31904 external_mini_cluster.cc:1428] Setting key a033ee0112868d07e40635a2c3765bfb
Sep 28 14:56:31 dist-test-slave-j968 krb5kdc[1883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607390, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.3@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:56:31.068284 2392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:31.047292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:49969 (local address 127.31.40.62:45209)
0928 14:56:31.047822 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:31.047827 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:31.049548 (+ 1721us) server_negotiation.cc:407] Connection header received
0928 14:56:31.051686 (+ 2138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:31.051690 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:31.051776 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:31.051970 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:31.053587 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:31.054477 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:31.056389 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:31.056803 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:31.056972 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:31.061000 (+ 4028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:31.061009 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:31.061015 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:31.061068 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:31.064628 (+ 3560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:31.065548 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:31.065553 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:31.065556 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:31.065703 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:31.066209 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:31.066212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:31.066215 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:31.066526 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:31.066655 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:31.067571 (+ 916us) server_negotiation.cc:299] Negotiation successful
0928 14:56:31.067760 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":271,"threads_started":1}
I20260928 14:56:31.069208 2393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:31.048000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:45209 (local address 127.31.40.3:49969)
0928 14:56:31.049351 (+ 1351us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:31.049409 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:31.051388 (+ 1979us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:31.052160 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:31.052169 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:31.052778 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:56:31.053389 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:31.053408 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:31.054692 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:31.054696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:31.056239 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:31.056253 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:31.057040 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:31.057047 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:31.057273 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:31.058069 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:56:31.058094 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:56:31.060697 (+ 2603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:56:31.064780 (+ 4083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:31.064789 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:31.064807 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:31.065378 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:31.065809 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:31.065812 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:31.065815 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:31.066083 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:31.066668 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:56:31.066675 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:56:31.067123 (+ 448us) client_negotiation.cc:769] Sending connection context
0928 14:56:31.067384 (+ 261us) client_negotiation.cc:240] Negotiation successful
0928 14:56:31.067839 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1002,"thread_start_us":792,"threads_started":1}
I20260928 14:56:31.070942 2390 heartbeater.cc:347] Connected to a master server at 127.31.40.62:45209
I20260928 14:56:31.071287 2390 heartbeater.cc:464] Registering TS with master...
I20260928 14:56:31.072266 2390 heartbeater.cc:511] Master 127.31.40.62:45209 requested a full tablet report, sending...
I20260928 14:56:31.074237 1929 ts_manager.cc:194] Registered new tserver with Master: 3cad188a79a3414fb7b1e2b879bd5aee (127.31.40.3:38451)
I20260928 14:56:31.075315 1929 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.3@KRBTEST.COM'} at 127.31.40.3:49969
I20260928 14:56:31.076552 31904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 14:56:31.109923 2392 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:56:31.107151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57468 (local address 127.31.40.62:45209)
0928 14:56:31.107339 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:31.107344 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:31.107948 (+ 604us) server_negotiation.cc:407] Connection header received
0928 14:56:31.108104 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:31.108108 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:31.108179 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:31.108278 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:31.109633 (+ 1355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57468: BlockingRecv error: recv got EOF from 127.0.0.1:57468 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 14:56:31.113519 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 1991
W20260928 14:56:31.116878 31904 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 1991: Not authorized: Client connection negotiation failed: client connection to 127.31.40.1:39679: server requires authentication, but client does not have Kerberos credentials available
I20260928 14:56:31.117195 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 1991
I20260928 14:56:31.134253 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 2126
W20260928 14:56:31.137555 31904 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 2126: Not authorized: Client connection negotiation failed: client connection to 127.31.40.2:36285: server requires authentication, but client does not have Kerberos credentials available
I20260928 14:56:31.137760 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 2126
I20260928 14:56:31.154066 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 2261
W20260928 14:56:31.157269 31904 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 2261: Not authorized: Client connection negotiation failed: client connection to 127.31.40.3:38451: server requires authentication, but client does not have Kerberos credentials available
I20260928 14:56:31.157433 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 2261
I20260928 14:56:31.173414 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 1896
I20260928 14:56:31.174726 1959 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:56:31.278806 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 1896
2026-09-28T14:56:31Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3801 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607365362981-31904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:56:31 dist-test-slave-j968 krb5kdc[2408](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:56:31 dist-test-slave-j968 krb5kdc[2408](info): set up 2 sockets
Sep 28 14:56:31 dist-test-slave-j968 krb5kdc[2408](info): commencing operation
krb5kdc: starting...
W20260928 14:56:32.450520 31904 mini_kdc.cc:121] Time spent starting KDC: real 1.125s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Sep 28 14:56:32 dist-test-slave-j968 krb5kdc[2408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:56:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:56:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab.
Entry for principal kudu/127.31.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab.
I20260928 14:56:32.667979 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607365362981-31904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607365362981-31904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607365362981-31904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:38063
--webserver_interface=127.31.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:41395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.40.62:38063
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:56:33.010460 2421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:56:33.010782 2421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:56:33.010869 2421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:56:33.018249 2421 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:56:33.018374 2421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:56:33.018421 2421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:56:33.018474 2421 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:56:33.018522 2421 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:56:33.028549 2421 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:41395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607365362981-31904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607365362981-31904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.40.62:38063
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:38063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607365362981-31904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:56:33.029855 2421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:56:33.031178 2421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:56:33.043448 2426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:33.043474 2427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:33.043612 2429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:33.044376 2421 server_base.cc:1034] running on GCE node
I20260928 14:56:33.045331 2421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:56:33.047015 2421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:56:33.048283 2421 hybrid_clock.cc:648] HybridClock initialized: now 1790607393048225 us; error 58 us; skew 500 ppm
I20260928 14:56:33.048853 2421 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:56:33 dist-test-slave-j968 krb5kdc[2408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607393, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:56:33.060086 2421 init.cc:377] Logged in from keytab as kudu/127.31.40.62@KRBTEST.COM (short username kudu)
I20260928 14:56:33.061838 2421 webserver.cc:460] Webserver started at http://127.31.40.62:34185/ using document root <none> and password file <none>
I20260928 14:56:33.062631 2421 fs_manager.cc:359] Metadata directory not provided
I20260928 14:56:33.062785 2421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:56:33.063118 2421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:56:33.066336 2421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607365362981-31904-0/minicluster-data/master-0/data/instance:
uuid: "2aac1cd2c570483fa3bbc28b30c699cf"
format_stamp: "Formatted at 2026-09-28 14:56:33 on dist-test-slave-j968"
server_key: "92d454420967ab539df2f18f4575219c"
server_key_iv: "888863ac1a3f8203547e480c8603592d"
server_key_version: "encryptionkey@0"
I20260928 14:56:33.067189 2421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607365362981-31904-0/minicluster-data/master-0/wal/instance:
uuid: "2aac1cd2c570483fa3bbc28b30c699cf"
format_stamp: "Formatted at 2026-09-28 14:56:33 on dist-test-slave-j968"
server_key: "92d454420967ab539df2f18f4575219c"
server_key_iv: "888863ac1a3f8203547e480c8603592d"
server_key_version: "encryptionkey@0"
I20260928 14:56:33.072723 2421 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260928 14:56:33.076583 2436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:33.078101 2421 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:56:33.078290 2421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607365362981-31904-0/minicluster-data/master-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607365362981-31904-0/minicluster-data/master-0/wal
uuid: "2aac1cd2c570483fa3bbc28b30c699cf"
format_stamp: "Formatted at 2026-09-28 14:56:33 on dist-test-slave-j968"
server_key: "92d454420967ab539df2f18f4575219c"
server_key_iv: "888863ac1a3f8203547e480c8603592d"
server_key_version: "encryptionkey@0"
I20260928 14:56:33.078491 2421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607365362981-31904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607365362981-31904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607365362981-31904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:56:33.099543 2421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:56:33.104573 2421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:56:33.104890 2421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:56:33.125015 2421 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.62:38063
I20260928 14:56:33.125062 2487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.62:38063 every 8 connection(s)
I20260928 14:56:33.126442 2421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
I20260928 14:56:33.129377 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 2421
I20260928 14:56:33.129640 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607365362981-31904-0/minicluster-data/master-0/wal/instance
I20260928 14:56:33.130565 31904 external_mini_cluster.cc:1428] Setting key b8fe7e68234d8179b7d8dba56f5f0bb6
I20260928 14:56:33.131346 2488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 14:56:33 dist-test-slave-j968 krb5kdc[2408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:56:33.145488 2488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2aac1cd2c570483fa3bbc28b30c699cf: Bootstrap starting.
I20260928 14:56:33.150452 2488 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2aac1cd2c570483fa3bbc28b30c699cf: Neither blocks nor log segments found. Creating new log.
I20260928 14:56:33.151922 2488 log.cc:824] T 00000000000000000000000000000000 P 2aac1cd2c570483fa3bbc28b30c699cf: Log is configured to *not* fsync() on all Append() calls
I20260928 14:56:33.155670 2488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2aac1cd2c570483fa3bbc28b30c699cf: No bootstrap required, opened a new log
I20260928 14:56:33.155941 2491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:33.132580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47526 (local address 127.31.40.62:38063)
0928 14:56:33.133393 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:33.133407 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:33.133449 (+ 42us) server_negotiation.cc:407] Connection header received
0928 14:56:33.135266 (+ 1817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:33.135305 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:33.135819 (+ 514us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:33.136354 (+ 535us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:33.137946 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:33.139359 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:33.140726 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:33.141448 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:33.141683 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:33.145992 (+ 4309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:33.146006 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:33.146030 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:33.146098 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:33.150711 (+ 4613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:33.151596 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:33.151608 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:33.151630 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:33.151922 (+ 292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:33.152369 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:33.152376 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:33.152380 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:33.153160 (+ 780us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:33.153467 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:33.153945 (+ 478us) server_negotiation.cc:299] Negotiation successful
0928 14:56:33.154428 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":365,"threads_started":1}
I20260928 14:56:33.160600 31904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:56:33.165185 2488 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2aac1cd2c570483fa3bbc28b30c699cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aac1cd2c570483fa3bbc28b30c699cf" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 38063 } }
I20260928 14:56:33.165632 2488 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2aac1cd2c570483fa3bbc28b30c699cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:56:33.165759 2488 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2aac1cd2c570483fa3bbc28b30c699cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aac1cd2c570483fa3bbc28b30c699cf, State: Initialized, Role: FOLLOWER
I20260928 14:56:33.166540 2488 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2aac1cd2c570483fa3bbc28b30c699cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aac1cd2c570483fa3bbc28b30c699cf" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 38063 } }
I20260928 14:56:33.166817 2488 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2aac1cd2c570483fa3bbc28b30c699cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:56:33.166962 2488 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2aac1cd2c570483fa3bbc28b30c699cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:56:33.167156 2488 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2aac1cd2c570483fa3bbc28b30c699cf [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:56:33.169610 2488 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2aac1cd2c570483fa3bbc28b30c699cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aac1cd2c570483fa3bbc28b30c699cf" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 38063 } }
I20260928 14:56:33.170220 2488 leader_election.cc:302] T 00000000000000000000000000000000 P 2aac1cd2c570483fa3bbc28b30c699cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2aac1cd2c570483fa3bbc28b30c699cf; no voters:
I20260928 14:56:33.171010 2488 leader_election.cc:288] T 00000000000000000000000000000000 P 2aac1cd2c570483fa3bbc28b30c699cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:56:33.171128 2494 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2aac1cd2c570483fa3bbc28b30c699cf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:56:33.172547 2494 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2aac1cd2c570483fa3bbc28b30c699cf [term 1 LEADER]: Becoming Leader. State: Replica: 2aac1cd2c570483fa3bbc28b30c699cf, State: Running, Role: LEADER
I20260928 14:56:33.173177 2494 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2aac1cd2c570483fa3bbc28b30c699cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aac1cd2c570483fa3bbc28b30c699cf" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 38063 } }
I20260928 14:56:33.173792 2488 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2aac1cd2c570483fa3bbc28b30c699cf [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:56:33.177407 2496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aac1cd2c570483fa3bbc28b30c699cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2aac1cd2c570483fa3bbc28b30c699cf. Latest consensus state: current_term: 1 leader_uuid: "2aac1cd2c570483fa3bbc28b30c699cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aac1cd2c570483fa3bbc28b30c699cf" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 38063 } } }
I20260928 14:56:33.177953 2496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aac1cd2c570483fa3bbc28b30c699cf [sys.catalog]: This master's current role is: LEADER
I20260928 14:56:33.190835 2495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aac1cd2c570483fa3bbc28b30c699cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2aac1cd2c570483fa3bbc28b30c699cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aac1cd2c570483fa3bbc28b30c699cf" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 38063 } } }
I20260928 14:56:33.191270 2495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aac1cd2c570483fa3bbc28b30c699cf [sys.catalog]: This master's current role is: LEADER
I20260928 14:56:33.192191 2502 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:56:33.197273 2502 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:56:33.209452 2502 catalog_manager.cc:1347] Generated new cluster ID: 596c334bf9934da6b023c5bde10a7b07
I20260928 14:56:33.209590 2502 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:56:33.211856 2491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:33.201817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47540 (local address 127.31.40.62:38063)
0928 14:56:33.202009 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:33.202015 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:33.202869 (+ 854us) server_negotiation.cc:407] Connection header received
0928 14:56:33.203067 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:33.203074 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:33.203182 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:33.203314 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:33.204608 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:33.205937 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:33.207452 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:33.207993 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:33.208147 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:33.209736 (+ 1589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:33.209744 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:33.209749 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:56:33.210738 (+ 989us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:33.210806 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:56:33.210849 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:33.210944 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:33.211269 (+ 325us) server_negotiation.cc:299] Negotiation successful
0928 14:56:33.211409 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 14:56:33.225718 2502 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:56:33.228086 2502 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:56:33.247218 2502 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2aac1cd2c570483fa3bbc28b30c699cf: Generated new TSK 0
I20260928 14:56:33.248340 2502 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:56:33.257093 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 2421
I20260928 14:56:33.258713 2483 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:56:33.353300 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 2421
2026-09-28T14:56:33Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2066 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:56:33 dist-test-slave-j968 krb5kdc[2523](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:56:33 dist-test-slave-j968 krb5kdc[2523](info): set up 2 sockets
Sep 28 14:56:33 dist-test-slave-j968 krb5kdc[2523](info): commencing operation
krb5kdc: starting...
W20260928 14:56:34.506632 31904 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Sep 28 14:56:34 dist-test-slave-j968 krb5kdc[2523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:56:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:56:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab.
Entry for principal kudu/127.31.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab.
I20260928 14:56:34.724520 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:33873
--webserver_interface=127.31.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:44481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.40.62:33873
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:56:35.055209 2536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:56:35.055621 2536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:56:35.055795 2536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:56:35.063278 2536 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:56:35.063449 2536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:56:35.063580 2536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:56:35.063688 2536 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:56:35.063772 2536 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:56:35.073997 2536 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:44481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.40.62:33873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:33873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:56:35.075531 2536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:56:35.077001 2536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:56:35.089789 2541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:35.089820 2544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:35.089764 2542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:35.090967 2536 server_base.cc:1034] running on GCE node
I20260928 14:56:35.091938 2536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:56:35.093624 2536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:56:35.094842 2536 hybrid_clock.cc:648] HybridClock initialized: now 1790607395094773 us; error 60 us; skew 500 ppm
I20260928 14:56:35.095216 2536 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:56:35 dist-test-slave-j968 krb5kdc[2523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607395, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:56:35.098433 2536 init.cc:377] Logged in from keytab as kudu/127.31.40.62@KRBTEST.COM (short username kudu)
I20260928 14:56:35.100149 2536 webserver.cc:460] Webserver started at http://127.31.40.62:42117/ using document root <none> and password file <none>
I20260928 14:56:35.100850 2536 fs_manager.cc:359] Metadata directory not provided
I20260928 14:56:35.100978 2536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:56:35.101308 2536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:56:35.104518 2536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/master-0/data/instance:
uuid: "bbbf392229894ed5b023f5ca98fec098"
format_stamp: "Formatted at 2026-09-28 14:56:35 on dist-test-slave-j968"
server_key: "0e03cbbf90780786a8d507a968dca2f4"
server_key_iv: "6a49f704d002791a1c61f5ec254b773e"
server_key_version: "encryptionkey@0"
I20260928 14:56:35.105294 2536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/master-0/wal/instance:
uuid: "bbbf392229894ed5b023f5ca98fec098"
format_stamp: "Formatted at 2026-09-28 14:56:35 on dist-test-slave-j968"
server_key: "0e03cbbf90780786a8d507a968dca2f4"
server_key_iv: "6a49f704d002791a1c61f5ec254b773e"
server_key_version: "encryptionkey@0"
I20260928 14:56:35.110507 2536 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 14:56:35.114401 2551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:35.116094 2536 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:56:35.116309 2536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/master-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/master-0/wal
uuid: "bbbf392229894ed5b023f5ca98fec098"
format_stamp: "Formatted at 2026-09-28 14:56:35 on dist-test-slave-j968"
server_key: "0e03cbbf90780786a8d507a968dca2f4"
server_key_iv: "6a49f704d002791a1c61f5ec254b773e"
server_key_version: "encryptionkey@0"
I20260928 14:56:35.116528 2536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:56:35.133364 2536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:56:35.137519 2536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:56:35.137825 2536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:56:35.156993 2536 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.62:33873
I20260928 14:56:35.157021 2602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.62:33873 every 8 connection(s)
I20260928 14:56:35.158497 2536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
I20260928 14:56:35.162444 2603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:56:35.165184 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 2536
I20260928 14:56:35.165419 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/master-0/wal/instance
I20260928 14:56:35.166095 31904 external_mini_cluster.cc:1428] Setting key 2429e195ba522dac82ff2d8342f688de
I20260928 14:56:35.175603 2603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbbf392229894ed5b023f5ca98fec098: Bootstrap starting.
I20260928 14:56:35.179286 2603 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbbf392229894ed5b023f5ca98fec098: Neither blocks nor log segments found. Creating new log.
I20260928 14:56:35.180372 2603 log.cc:824] T 00000000000000000000000000000000 P bbbf392229894ed5b023f5ca98fec098: Log is configured to *not* fsync() on all Append() calls
Sep 28 14:56:35 dist-test-slave-j968 krb5kdc[2523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:56:35.183807 2603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbbf392229894ed5b023f5ca98fec098: No bootstrap required, opened a new log
I20260928 14:56:35.192869 2606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:35.168311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52262 (local address 127.31.40.62:33873)
0928 14:56:35.169090 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:35.169106 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:35.169154 (+ 48us) server_negotiation.cc:407] Connection header received
0928 14:56:35.171372 (+ 2218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:35.171403 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:35.171974 (+ 571us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:35.172425 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:35.173729 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:35.175681 (+ 1952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:35.177299 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:35.178329 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:35.178518 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:35.183155 (+ 4637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:35.183167 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:35.183187 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:35.183237 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:35.187448 (+ 4211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:35.188413 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:35.188424 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:35.188440 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:35.188690 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:35.189150 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:35.189155 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:35.189158 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:35.190093 (+ 935us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:35.190428 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:35.190848 (+ 420us) server_negotiation.cc:299] Negotiation successful
0928 14:56:35.191355 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":329,"threads_started":1}
I20260928 14:56:35.194651 2603 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbbf392229894ed5b023f5ca98fec098 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbbf392229894ed5b023f5ca98fec098" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 33873 } }
I20260928 14:56:35.195020 2603 raft_consensus.cc:374] T 00000000000000000000000000000000 P bbbf392229894ed5b023f5ca98fec098 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:56:35.195150 2603 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbbf392229894ed5b023f5ca98fec098 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbbf392229894ed5b023f5ca98fec098, State: Initialized, Role: FOLLOWER
I20260928 14:56:35.195732 2603 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbbf392229894ed5b023f5ca98fec098 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbbf392229894ed5b023f5ca98fec098" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 33873 } }
I20260928 14:56:35.196054 2603 raft_consensus.cc:388] T 00000000000000000000000000000000 P bbbf392229894ed5b023f5ca98fec098 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:56:35.196209 2603 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbbf392229894ed5b023f5ca98fec098 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:56:35.196408 2603 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbbf392229894ed5b023f5ca98fec098 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:56:35.198709 2603 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbbf392229894ed5b023f5ca98fec098 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbbf392229894ed5b023f5ca98fec098" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 33873 } }
I20260928 14:56:35.199322 2603 leader_election.cc:302] T 00000000000000000000000000000000 P bbbf392229894ed5b023f5ca98fec098 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bbbf392229894ed5b023f5ca98fec098; no voters:
I20260928 14:56:35.200158 2603 leader_election.cc:288] T 00000000000000000000000000000000 P bbbf392229894ed5b023f5ca98fec098 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:56:35.200304 2608 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbbf392229894ed5b023f5ca98fec098 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:56:35.201030 2608 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbbf392229894ed5b023f5ca98fec098 [term 1 LEADER]: Becoming Leader. State: Replica: bbbf392229894ed5b023f5ca98fec098, State: Running, Role: LEADER
I20260928 14:56:35.201635 2608 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbbf392229894ed5b023f5ca98fec098 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbbf392229894ed5b023f5ca98fec098" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 33873 } }
I20260928 14:56:35.202368 2603 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbbf392229894ed5b023f5ca98fec098 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:56:35.204437 2609 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbbf392229894ed5b023f5ca98fec098 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bbbf392229894ed5b023f5ca98fec098" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbbf392229894ed5b023f5ca98fec098" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 33873 } } }
I20260928 14:56:35.204804 2609 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbbf392229894ed5b023f5ca98fec098 [sys.catalog]: This master's current role is: LEADER
I20260928 14:56:35.205971 2610 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbbf392229894ed5b023f5ca98fec098 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbbf392229894ed5b023f5ca98fec098. Latest consensus state: current_term: 1 leader_uuid: "bbbf392229894ed5b023f5ca98fec098" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbbf392229894ed5b023f5ca98fec098" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 33873 } } }
I20260928 14:56:35.206439 2610 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbbf392229894ed5b023f5ca98fec098 [sys.catalog]: This master's current role is: LEADER
I20260928 14:56:35.207409 2614 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:56:35.212862 2614 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:56:35.222234 2614 catalog_manager.cc:1347] Generated new cluster ID: 3f132a37fe7b430c96d1051a90da9df7
I20260928 14:56:35.222375 2614 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:56:35.247805 2614 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:56:35.249226 2614 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:56:35.255960 2614 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bbbf392229894ed5b023f5ca98fec098: Generated new TSK 0
I20260928 14:56:35.256884 2614 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.40.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab.
Entry for principal kudu/127.31.40.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab.
I20260928 14:56:35.312922 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.1:0
--local_ip_for_outbound_sockets=127.31.40.1
--webserver_interface=127.31.40.1
--webserver_port=0
--tserver_master_addrs=127.31.40.62:33873
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab
--principal=kudu/127.31.40.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:44481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:56:35.654016 2629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:56:35.654314 2629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:56:35.654409 2629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:56:35.661729 2629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:56:35.661854 2629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:56:35.661994 2629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.1
I20260928 14:56:35.671835 2629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:44481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab
--principal=kudu/127.31.40.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.40.1
--webserver_port=0
--tserver_master_addrs=127.31.40.62:33873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.1
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:56:35.673164 2629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:56:35.674590 2629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:56:35.688623 2634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:35.688606 2637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:35.688565 2635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:35.689313 2629 server_base.cc:1034] running on GCE node
I20260928 14:56:35.689949 2629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:56:35.690922 2629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:56:35.692142 2629 hybrid_clock.cc:648] HybridClock initialized: now 1790607395692067 us; error 60 us; skew 500 ppm
I20260928 14:56:35.692497 2629 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:56:35 dist-test-slave-j968 krb5kdc[2523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607395, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:56:35.695796 2629 init.cc:377] Logged in from keytab as kudu/127.31.40.1@KRBTEST.COM (short username kudu)
I20260928 14:56:35.697376 2629 webserver.cc:460] Webserver started at http://127.31.40.1:37563/ using document root <none> and password file <none>
I20260928 14:56:35.698045 2629 fs_manager.cc:359] Metadata directory not provided
I20260928 14:56:35.698144 2629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:56:35.698385 2629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:56:35.702420 2629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-0/data/instance:
uuid: "49d9868f823e424893a316a35e1b8d2f"
format_stamp: "Formatted at 2026-09-28 14:56:35 on dist-test-slave-j968"
server_key: "5f0956edbc268715e31b63d40fa4737b"
server_key_iv: "bd5bc05e5bf0198b2dd2432b8db80f93"
server_key_version: "encryptionkey@0"
I20260928 14:56:35.703298 2629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-0/wal/instance:
uuid: "49d9868f823e424893a316a35e1b8d2f"
format_stamp: "Formatted at 2026-09-28 14:56:35 on dist-test-slave-j968"
server_key: "5f0956edbc268715e31b63d40fa4737b"
server_key_iv: "bd5bc05e5bf0198b2dd2432b8db80f93"
server_key_version: "encryptionkey@0"
I20260928 14:56:35.708932 2629 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260928 14:56:35.713043 2644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:35.714555 2629 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 14:56:35.714751 2629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-0/wal
uuid: "49d9868f823e424893a316a35e1b8d2f"
format_stamp: "Formatted at 2026-09-28 14:56:35 on dist-test-slave-j968"
server_key: "5f0956edbc268715e31b63d40fa4737b"
server_key_iv: "bd5bc05e5bf0198b2dd2432b8db80f93"
server_key_version: "encryptionkey@0"
I20260928 14:56:35.714985 2629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:56:35.739802 2629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:56:35.744596 2629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:56:35.744912 2629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:56:35.746021 2629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:56:35.747435 2629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:56:35.747541 2629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:35.747689 2629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:56:35.747761 2629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:35.782685 2629 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.1:40081
I20260928 14:56:35.782787 2756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.1:40081 every 8 connection(s)
I20260928 14:56:35.784204 2629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
I20260928 14:56:35.793627 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 2629
I20260928 14:56:35.793825 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-0/wal/instance
I20260928 14:56:35.794463 31904 external_mini_cluster.cc:1428] Setting key 75237cc7960cad3fc93149fe258e5951
Sep 28 14:56:35 dist-test-slave-j968 krb5kdc[2523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607395, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.1@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:56:35.804461 2759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:35.786839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:44661 (local address 127.31.40.62:33873)
0928 14:56:35.787310 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:35.787314 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:35.788353 (+ 1039us) server_negotiation.cc:407] Connection header received
0928 14:56:35.790208 (+ 1855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:35.790214 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:35.790310 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:35.790431 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:35.792077 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:35.792821 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:35.794512 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:35.794953 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:35.795089 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:35.798581 (+ 3492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:35.798586 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:35.798589 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:35.798621 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:35.801224 (+ 2603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:35.801993 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:35.802006 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:35.802008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:35.802121 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:35.802626 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:35.802633 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:35.802641 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:35.802922 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:35.803078 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:35.803817 (+ 739us) server_negotiation.cc:299] Negotiation successful
0928 14:56:35.804072 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":244,"threads_started":1}
I20260928 14:56:35.805490 2760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:35.787439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:33873 (local address 127.31.40.1:44661)
0928 14:56:35.788195 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:35.788233 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:35.789938 (+ 1705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:35.790674 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:35.790683 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:35.791265 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:56:35.791837 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:35.791859 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:35.792974 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:35.792980 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:35.794345 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:35.794357 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:35.795190 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:35.795197 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:35.795433 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:35.796209 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:56:35.796246 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:56:35.798346 (+ 2100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:56:35.801379 (+ 3033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:35.801388 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:35.801413 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:35.801844 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:35.802241 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:35.802245 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:35.802249 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:35.802467 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:35.803053 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:56:35.803061 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:56:35.803419 (+ 358us) client_negotiation.cc:769] Sending connection context
0928 14:56:35.803737 (+ 318us) client_negotiation.cc:240] Negotiation successful
0928 14:56:35.804319 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":271,"threads_started":1}
I20260928 14:56:35.807210 2757 heartbeater.cc:347] Connected to a master server at 127.31.40.62:33873
I20260928 14:56:35.807569 2757 heartbeater.cc:464] Registering TS with master...
I20260928 14:56:35.808600 2757 heartbeater.cc:511] Master 127.31.40.62:33873 requested a full tablet report, sending...
I20260928 14:56:35.811190 2568 ts_manager.cc:194] Registered new tserver with Master: 49d9868f823e424893a316a35e1b8d2f (127.31.40.1:40081)
WARNING: no policy specified for kudu/127.31.40.2@KRBTEST.COM; defaulting to no policy
I20260928 14:56:35.813467 2568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.1@KRBTEST.COM'} at 127.31.40.1:44661
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab.
Entry for principal kudu/127.31.40.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab.
I20260928 14:56:35.843418 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.2:0
--local_ip_for_outbound_sockets=127.31.40.2
--webserver_interface=127.31.40.2
--webserver_port=0
--tserver_master_addrs=127.31.40.62:33873
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab
--principal=kudu/127.31.40.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:44481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:56:36.191521 2763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:56:36.191965 2763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:56:36.192168 2763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:56:36.200593 2763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:56:36.200726 2763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:56:36.200899 2763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.2
I20260928 14:56:36.211237 2763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:44481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab
--principal=kudu/127.31.40.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.40.2
--webserver_port=0
--tserver_master_addrs=127.31.40.62:33873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.2
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:56:36.212777 2763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:56:36.214141 2763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:56:36.228822 2772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:36.228689 2769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:36.228736 2770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:36.229620 2763 server_base.cc:1034] running on GCE node
I20260928 14:56:36.230309 2763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:56:36.231564 2763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:56:36.232769 2763 hybrid_clock.cc:648] HybridClock initialized: now 1790607396232690 us; error 73 us; skew 500 ppm
I20260928 14:56:36.233127 2763 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:56:36 dist-test-slave-j968 krb5kdc[2523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607396, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:56:36.236351 2763 init.cc:377] Logged in from keytab as kudu/127.31.40.2@KRBTEST.COM (short username kudu)
I20260928 14:56:36.237926 2763 webserver.cc:460] Webserver started at http://127.31.40.2:40867/ using document root <none> and password file <none>
I20260928 14:56:36.238664 2763 fs_manager.cc:359] Metadata directory not provided
I20260928 14:56:36.238792 2763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:56:36.239114 2763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:56:36.242206 2763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-1/data/instance:
uuid: "40185e92c61e4a45ad03c4f44cd6dbeb"
format_stamp: "Formatted at 2026-09-28 14:56:36 on dist-test-slave-j968"
server_key: "411eb8e7bf16cd5191a0faf2b5231030"
server_key_iv: "4e345789dda9d4190913086bcc83caf5"
server_key_version: "encryptionkey@0"
I20260928 14:56:36.243032 2763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-1/wal/instance:
uuid: "40185e92c61e4a45ad03c4f44cd6dbeb"
format_stamp: "Formatted at 2026-09-28 14:56:36 on dist-test-slave-j968"
server_key: "411eb8e7bf16cd5191a0faf2b5231030"
server_key_iv: "4e345789dda9d4190913086bcc83caf5"
server_key_version: "encryptionkey@0"
I20260928 14:56:36.248471 2763 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 14:56:36.252359 2779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:36.253975 2763 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:56:36.254176 2763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-1/wal
uuid: "40185e92c61e4a45ad03c4f44cd6dbeb"
format_stamp: "Formatted at 2026-09-28 14:56:36 on dist-test-slave-j968"
server_key: "411eb8e7bf16cd5191a0faf2b5231030"
server_key_iv: "4e345789dda9d4190913086bcc83caf5"
server_key_version: "encryptionkey@0"
I20260928 14:56:36.254391 2763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:56:36.284999 2763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:56:36.289655 2763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:56:36.289992 2763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:56:36.291013 2763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:56:36.292508 2763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:56:36.292582 2763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:36.292672 2763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:56:36.292718 2763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:36.330595 2763 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.2:37865
I20260928 14:56:36.330650 2892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.2:37865 every 8 connection(s)
I20260928 14:56:36.332262 2763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
I20260928 14:56:36.334925 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 2763
I20260928 14:56:36.335129 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-1/wal/instance
I20260928 14:56:36.335676 31904 external_mini_cluster.cc:1428] Setting key 6b3492cd953ce77bbb8ad0d89f093a1a
Sep 28 14:56:36 dist-test-slave-j968 krb5kdc[2523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607396, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.2@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.40.3@KRBTEST.COM; defaulting to no policy
I20260928 14:56:36.358619 2895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:36.335421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:42397 (local address 127.31.40.62:33873)
0928 14:56:36.335950 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:36.335958 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:36.337286 (+ 1328us) server_negotiation.cc:407] Connection header received
0928 14:56:36.340029 (+ 2743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:36.340036 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:36.340112 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:36.340221 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:36.342576 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:36.343311 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:36.345330 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:36.346133 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:36.346294 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:36.350265 (+ 3971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:36.350276 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:36.350283 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:36.350334 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:36.354745 (+ 4411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:36.355683 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:36.355691 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:36.355697 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:36.355902 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:36.356450 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:36.356459 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:36.356463 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:36.356841 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:36.357041 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:36.357989 (+ 948us) server_negotiation.cc:299] Negotiation successful
0928 14:56:36.358226 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":241,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.3@KRBTEST.COM" created.
I20260928 14:56:36.360014 2896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:36.336198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:33873 (local address 127.31.40.2:42397)
0928 14:56:36.337124 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:36.337172 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:36.339660 (+ 2488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:36.340540 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:36.340556 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:36.341398 (+ 842us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:56:36.342335 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:36.342364 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:36.343488 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:36.343497 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:36.345161 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:36.345175 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:36.346425 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:36.346436 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:36.346795 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:36.347640 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:56:36.347665 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:56:36.349983 (+ 2318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:56:36.354954 (+ 4971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:36.354963 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:36.354980 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:36.355511 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:36.356068 (+ 557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:36.356073 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:36.356076 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:36.356294 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:36.357060 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:56:36.357071 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:56:36.357509 (+ 438us) client_negotiation.cc:769] Sending connection context
0928 14:56:36.357819 (+ 310us) client_negotiation.cc:240] Negotiation successful
0928 14:56:36.358420 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":407,"threads_started":1}
I20260928 14:56:36.361730 2893 heartbeater.cc:347] Connected to a master server at 127.31.40.62:33873
I20260928 14:56:36.362133 2893 heartbeater.cc:464] Registering TS with master...
I20260928 14:56:36.363193 2893 heartbeater.cc:511] Master 127.31.40.62:33873 requested a full tablet report, sending...
I20260928 14:56:36.365191 2568 ts_manager.cc:194] Registered new tserver with Master: 40185e92c61e4a45ad03c4f44cd6dbeb (127.31.40.2:37865)
I20260928 14:56:36.366391 2568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.2@KRBTEST.COM'} at 127.31.40.2:42397
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab.
Entry for principal kudu/127.31.40.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab.
I20260928 14:56:36.387836 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.3:0
--local_ip_for_outbound_sockets=127.31.40.3
--webserver_interface=127.31.40.3
--webserver_port=0
--tserver_master_addrs=127.31.40.62:33873
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab
--principal=kudu/127.31.40.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:44481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:56:36.732817 2899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:56:36.733137 2899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:56:36.733232 2899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:56:36.740795 2899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:56:36.740926 2899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:56:36.741048 2899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.3
I20260928 14:56:36.751410 2899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:44481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab
--principal=kudu/127.31.40.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.40.3
--webserver_port=0
--tserver_master_addrs=127.31.40.62:33873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.3
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:56:36.752938 2899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:56:36.754679 2899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:56:36.769412 2904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:36.769375 2905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:36.769452 2907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:36.771278 2899 server_base.cc:1034] running on GCE node
I20260928 14:56:36.772083 2899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:56:36.773165 2899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:56:36.774420 2899 hybrid_clock.cc:648] HybridClock initialized: now 1790607396774347 us; error 65 us; skew 500 ppm
I20260928 14:56:36.774775 2899 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:56:36 dist-test-slave-j968 krb5kdc[2523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607396, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:56:36.778242 2899 init.cc:377] Logged in from keytab as kudu/127.31.40.3@KRBTEST.COM (short username kudu)
I20260928 14:56:36.779939 2899 webserver.cc:460] Webserver started at http://127.31.40.3:40849/ using document root <none> and password file <none>
I20260928 14:56:36.780707 2899 fs_manager.cc:359] Metadata directory not provided
I20260928 14:56:36.780858 2899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:56:36.781193 2899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:56:36.785315 2899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-2/data/instance:
uuid: "5b66e7342b1f4f66a8c609f92cdc5f78"
format_stamp: "Formatted at 2026-09-28 14:56:36 on dist-test-slave-j968"
server_key: "aab2d3615e51c1abd1035c90ce45b63d"
server_key_iv: "6b27f08084bb27af93ebdbe1de1f9a14"
server_key_version: "encryptionkey@0"
I20260928 14:56:36.786377 2899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-2/wal/instance:
uuid: "5b66e7342b1f4f66a8c609f92cdc5f78"
format_stamp: "Formatted at 2026-09-28 14:56:36 on dist-test-slave-j968"
server_key: "aab2d3615e51c1abd1035c90ce45b63d"
server_key_iv: "6b27f08084bb27af93ebdbe1de1f9a14"
server_key_version: "encryptionkey@0"
I20260928 14:56:36.792505 2899 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260928 14:56:36.795950 2914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:36.797418 2899 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:56:36.797611 2899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-2/wal
uuid: "5b66e7342b1f4f66a8c609f92cdc5f78"
format_stamp: "Formatted at 2026-09-28 14:56:36 on dist-test-slave-j968"
server_key: "aab2d3615e51c1abd1035c90ce45b63d"
server_key_iv: "6b27f08084bb27af93ebdbe1de1f9a14"
server_key_version: "encryptionkey@0"
I20260928 14:56:36.797816 2899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:56:36.811766 2899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:56:36.815958 2899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:56:36.816282 2899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:56:36.817176 2757 heartbeater.cc:503] Master 127.31.40.62:33873 was elected leader, sending a full tablet report...
I20260928 14:56:36.817399 2899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:56:36.819228 2899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:56:36.819339 2899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:36.819473 2899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:56:36.819547 2899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:36.853952 2899 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.3:39703
I20260928 14:56:36.853992 3027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.3:39703 every 8 connection(s)
I20260928 14:56:36.855310 2899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
I20260928 14:56:36.858839 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 2899
I20260928 14:56:36.859022 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607365362981-31904-0/minicluster-data/ts-2/wal/instance
I20260928 14:56:36.859726 31904 external_mini_cluster.cc:1428] Setting key 8098f94b747beb81fb2976bae46f9c17
Sep 28 14:56:36 dist-test-slave-j968 krb5kdc[2523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607396, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.3@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:56:36.877417 2895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:36.858258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:34743 (local address 127.31.40.62:33873)
0928 14:56:36.858537 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:36.858544 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:36.860005 (+ 1461us) server_negotiation.cc:407] Connection header received
0928 14:56:36.862630 (+ 2625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:36.862636 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:36.862724 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:36.862833 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:36.864709 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:36.865471 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:36.867079 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:36.867459 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:36.867566 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:36.871008 (+ 3442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:36.871014 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:36.871017 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:36.871054 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:36.873877 (+ 2823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:36.874958 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:36.874964 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:36.874968 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:36.875172 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:36.875665 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:36.875672 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:36.875678 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:36.876141 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:36.876263 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:36.876973 (+ 710us) server_negotiation.cc:299] Negotiation successful
0928 14:56:36.877193 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260928 14:56:36.878404 3030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:36.858861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:33873 (local address 127.31.40.3:34743)
0928 14:56:36.859797 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:36.859849 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:36.862259 (+ 2410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:36.863190 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:36.863205 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:36.863814 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:56:36.864527 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:36.864547 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:36.865645 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:36.865649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:36.866929 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:36.866941 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:36.867666 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:36.867671 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:36.867935 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:36.868591 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:56:36.868617 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:56:36.870768 (+ 2151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:56:36.874138 (+ 3370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:36.874149 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:36.874169 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:36.874804 (+ 635us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:36.875297 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:36.875303 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:36.875306 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:36.875532 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:36.876265 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:56:36.876272 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:56:36.876624 (+ 352us) client_negotiation.cc:769] Sending connection context
0928 14:56:36.876862 (+ 238us) client_negotiation.cc:240] Negotiation successful
0928 14:56:36.877293 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":360,"threads_started":1}
I20260928 14:56:36.879812 3028 heartbeater.cc:347] Connected to a master server at 127.31.40.62:33873
I20260928 14:56:36.880229 3028 heartbeater.cc:464] Registering TS with master...
I20260928 14:56:36.880995 3028 heartbeater.cc:511] Master 127.31.40.62:33873 requested a full tablet report, sending...
I20260928 14:56:36.882465 2568 ts_manager.cc:194] Registered new tserver with Master: 5b66e7342b1f4f66a8c609f92cdc5f78 (127.31.40.3:39703)
I20260928 14:56:36.883291 2568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.3@KRBTEST.COM'} at 127.31.40.3:34743
I20260928 14:56:36.888866 31904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 14:56:36 dist-test-slave-j968 krb5kdc[2523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607396, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 14:56:36 dist-test-slave-j968 krb5kdc[2523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607396, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:56:36.940225 2895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:36.925692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52270 (local address 127.31.40.62:33873)
0928 14:56:36.925887 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:36.925892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:36.926446 (+ 554us) server_negotiation.cc:407] Connection header received
0928 14:56:36.926560 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:36.926565 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:36.926649 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:36.926755 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:36.928269 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:36.929030 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:36.930375 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:36.930796 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:36.930915 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:36.935158 (+ 4243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:36.935164 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:36.935167 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:36.935209 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:36.937814 (+ 2605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:36.938465 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:36.938469 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:36.938471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:36.938601 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:36.938932 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:36.938935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:36.938937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:36.939254 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:36.939444 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:36.939663 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 14:56:36.939844 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260928 14:56:36.940814 2568 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52270
I20260928 14:56:36.942263 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 2629
Sep 28 14:56:36 dist-test-slave-j968 krb5kdc[2523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607396, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.31.40.1@KRBTEST.COM
I20260928 14:56:36.959422 3039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:36.943502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44580 (local address 127.31.40.1:40081)
0928 14:56:36.944257 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:36.944265 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:36.944291 (+ 26us) server_negotiation.cc:407] Connection header received
0928 14:56:36.944356 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:36.944363 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:36.944590 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:36.944734 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:36.945922 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:36.947105 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:36.948668 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:36.949255 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:36.949362 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:36.953538 (+ 4176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:36.953552 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:36.953569 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:36.953605 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:36.956431 (+ 2826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:36.957172 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:36.957180 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:36.957183 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:36.957308 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:36.957746 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:36.957753 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:36.957757 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:36.958119 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:36.958380 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:36.958737 (+ 357us) server_negotiation.cc:299] Negotiation successful
0928 14:56:36.959068 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":381,"threads_started":1}
W20260928 14:56:36.960857 2752 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:44580
W20260928 14:56:36.962069 31904 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 2629: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 14:56:36.962302 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 2629
I20260928 14:56:36.979197 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 2763
Sep 28 14:56:36 dist-test-slave-j968 krb5kdc[2523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607396, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.31.40.2@KRBTEST.COM
I20260928 14:56:36.996196 3040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:36.980147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37682 (local address 127.31.40.2:37865)
0928 14:56:36.980687 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:36.980694 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:36.980720 (+ 26us) server_negotiation.cc:407] Connection header received
0928 14:56:36.980788 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:36.980794 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:36.981004 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:36.981143 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:36.982533 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:36.983672 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:36.985086 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:36.985724 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:36.985835 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:36.990144 (+ 4309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:36.990157 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:36.990202 (+ 45us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:36.990256 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:36.993184 (+ 2928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:36.993835 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:36.993847 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:36.993849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:36.994070 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:36.994520 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:36.994524 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:36.994526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:36.994872 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:36.995145 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:36.995441 (+ 296us) server_negotiation.cc:299] Negotiation successful
0928 14:56:36.995696 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":271,"threads_started":1}
W20260928 14:56:36.997607 2888 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:37682
W20260928 14:56:36.998729 31904 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 2763: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 14:56:36.998868 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 2763
I20260928 14:56:37.015996 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 2899
Sep 28 14:56:37 dist-test-slave-j968 krb5kdc[2523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607396, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.31.40.3@KRBTEST.COM
I20260928 14:56:37.032970 3041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:37.017071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41944 (local address 127.31.40.3:39703)
0928 14:56:37.017833 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:37.017839 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:37.017870 (+ 31us) server_negotiation.cc:407] Connection header received
0928 14:56:37.017929 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:37.017935 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:37.018215 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:37.018346 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:37.019667 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:37.020894 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:37.022305 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:37.022875 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:37.022977 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:37.027302 (+ 4325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:37.027314 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:37.027330 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:37.027367 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:37.030286 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:37.030939 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:37.030946 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:37.030948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:37.031084 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:37.031471 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:37.031475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:37.031477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:37.031827 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:37.032196 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:37.032451 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 14:56:37.032680 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":407,"threads_started":1}
W20260928 14:56:37.034379 3023 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:41944
W20260928 14:56:37.035480 31904 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 2899: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 14:56:37.035671 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 2899
I20260928 14:56:37.051611 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 2536
I20260928 14:56:37.052979 2598 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:56:37.158416 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 2536
2026-09-28T14:56:37Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3809 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:56:37 dist-test-slave-j968 krb5kdc[3045](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:56:37 dist-test-slave-j968 krb5kdc[3045](info): set up 2 sockets
Sep 28 14:56:37 dist-test-slave-j968 krb5kdc[3045](info): commencing operation
krb5kdc: starting...
W20260928 14:56:38.262065 31904 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Sep 28 14:56:38 dist-test-slave-j968 krb5kdc[3045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:56:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:56:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab.
Entry for principal kudu/127.31.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab.
I20260928 14:56:38.479795 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:46865
--webserver_interface=127.31.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:38561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.40.62:46865
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:56:38.829310 3059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:56:38.829633 3059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:56:38.829809 3059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:56:38.837337 3059 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:56:38.837463 3059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:56:38.837580 3059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:56:38.837692 3059 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:56:38.837783 3059 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:56:38.848079 3059 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:38561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.40.62:46865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:46865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:56:38.849555 3059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:56:38.851017 3059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:56:38.863759 3064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:38.863842 3067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:38.863788 3065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:38.864619 3059 server_base.cc:1034] running on GCE node
I20260928 14:56:38.865541 3059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:56:38.867332 3059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:56:38.868623 3059 hybrid_clock.cc:648] HybridClock initialized: now 1790607398868519 us; error 86 us; skew 500 ppm
I20260928 14:56:38.869107 3059 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:56:38 dist-test-slave-j968 krb5kdc[3045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607398, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:56:38.877581 3059 init.cc:377] Logged in from keytab as kudu/127.31.40.62@KRBTEST.COM (short username kudu)
I20260928 14:56:38.879613 3059 webserver.cc:460] Webserver started at http://127.31.40.62:42065/ using document root <none> and password file <none>
I20260928 14:56:38.880487 3059 fs_manager.cc:359] Metadata directory not provided
I20260928 14:56:38.880620 3059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:56:38.880918 3059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:56:38.884502 3059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/master-0/data/instance:
uuid: "d364c65f3538400782afe33c299027cd"
format_stamp: "Formatted at 2026-09-28 14:56:38 on dist-test-slave-j968"
server_key: "a57dd521885b862c275a06feb542718c"
server_key_iv: "694e5cc422ce1fc2d4103b86822a72dd"
server_key_version: "encryptionkey@0"
I20260928 14:56:38.885509 3059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/master-0/wal/instance:
uuid: "d364c65f3538400782afe33c299027cd"
format_stamp: "Formatted at 2026-09-28 14:56:38 on dist-test-slave-j968"
server_key: "a57dd521885b862c275a06feb542718c"
server_key_iv: "694e5cc422ce1fc2d4103b86822a72dd"
server_key_version: "encryptionkey@0"
I20260928 14:56:38.891741 3059 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260928 14:56:38.896077 3074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:38.897761 3059 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 14:56:38.897971 3059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/master-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/master-0/wal
uuid: "d364c65f3538400782afe33c299027cd"
format_stamp: "Formatted at 2026-09-28 14:56:38 on dist-test-slave-j968"
server_key: "a57dd521885b862c275a06feb542718c"
server_key_iv: "694e5cc422ce1fc2d4103b86822a72dd"
server_key_version: "encryptionkey@0"
I20260928 14:56:38.898253 3059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:56:38.918107 3059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:56:38.922911 3059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:56:38.923215 3059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:56:38.943709 3059 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.62:46865
I20260928 14:56:38.943751 3125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.62:46865 every 8 connection(s)
I20260928 14:56:38.945190 3059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
I20260928 14:56:38.949208 3126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:56:38.951407 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 3059
I20260928 14:56:38.951653 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/master-0/wal/instance
I20260928 14:56:38.952484 31904 external_mini_cluster.cc:1428] Setting key 8f57ff0ba271ac060d702cd49f685ba6
I20260928 14:56:38.961910 3126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d364c65f3538400782afe33c299027cd: Bootstrap starting.
I20260928 14:56:38.965188 3126 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d364c65f3538400782afe33c299027cd: Neither blocks nor log segments found. Creating new log.
I20260928 14:56:38.966254 3126 log.cc:824] T 00000000000000000000000000000000 P d364c65f3538400782afe33c299027cd: Log is configured to *not* fsync() on all Append() calls
Sep 28 14:56:38 dist-test-slave-j968 krb5kdc[3045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:56:38.969329 3126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d364c65f3538400782afe33c299027cd: No bootstrap required, opened a new log
I20260928 14:56:38.976847 3129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:38.954769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44704 (local address 127.31.40.62:46865)
0928 14:56:38.955630 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:38.955642 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:38.955688 (+ 46us) server_negotiation.cc:407] Connection header received
0928 14:56:38.957466 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:38.957490 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:38.957918 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:38.958312 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:38.959817 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:38.961233 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:38.962587 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:38.963301 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:38.963516 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:38.967701 (+ 4185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:38.967709 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:38.967726 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:38.967764 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:38.971600 (+ 3836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:38.972486 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:38.972495 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:38.972507 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:38.972759 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:38.973128 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:38.973133 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:38.973136 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:38.973942 (+ 806us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:38.974255 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:38.974747 (+ 492us) server_negotiation.cc:299] Negotiation successful
0928 14:56:38.975288 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":344,"threads_started":1}
I20260928 14:56:38.980284 3126 raft_consensus.cc:348] T 00000000000000000000000000000000 P d364c65f3538400782afe33c299027cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d364c65f3538400782afe33c299027cd" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 46865 } }
I20260928 14:56:38.980705 3126 raft_consensus.cc:374] T 00000000000000000000000000000000 P d364c65f3538400782afe33c299027cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:56:38.980788 3126 raft_consensus.cc:729] T 00000000000000000000000000000000 P d364c65f3538400782afe33c299027cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d364c65f3538400782afe33c299027cd, State: Initialized, Role: FOLLOWER
I20260928 14:56:38.981359 3126 consensus_queue.cc:260] T 00000000000000000000000000000000 P d364c65f3538400782afe33c299027cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d364c65f3538400782afe33c299027cd" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 46865 } }
I20260928 14:56:38.981573 3126 raft_consensus.cc:388] T 00000000000000000000000000000000 P d364c65f3538400782afe33c299027cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:56:38.981670 3126 raft_consensus.cc:482] T 00000000000000000000000000000000 P d364c65f3538400782afe33c299027cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:56:38.981801 3126 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d364c65f3538400782afe33c299027cd [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:56:38.984020 3126 raft_consensus.cc:504] T 00000000000000000000000000000000 P d364c65f3538400782afe33c299027cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d364c65f3538400782afe33c299027cd" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 46865 } }
I20260928 14:56:38.984480 3126 leader_election.cc:302] T 00000000000000000000000000000000 P d364c65f3538400782afe33c299027cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d364c65f3538400782afe33c299027cd; no voters:
I20260928 14:56:38.985131 3126 leader_election.cc:288] T 00000000000000000000000000000000 P d364c65f3538400782afe33c299027cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:56:38.985289 3131 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d364c65f3538400782afe33c299027cd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:56:38.986455 3131 raft_consensus.cc:686] T 00000000000000000000000000000000 P d364c65f3538400782afe33c299027cd [term 1 LEADER]: Becoming Leader. State: Replica: d364c65f3538400782afe33c299027cd, State: Running, Role: LEADER
I20260928 14:56:38.987327 3131 consensus_queue.cc:237] T 00000000000000000000000000000000 P d364c65f3538400782afe33c299027cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d364c65f3538400782afe33c299027cd" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 46865 } }
I20260928 14:56:38.987426 3126 sys_catalog.cc:564] T 00000000000000000000000000000000 P d364c65f3538400782afe33c299027cd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:56:38.990319 3133 sys_catalog.cc:455] T 00000000000000000000000000000000 P d364c65f3538400782afe33c299027cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader d364c65f3538400782afe33c299027cd. Latest consensus state: current_term: 1 leader_uuid: "d364c65f3538400782afe33c299027cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d364c65f3538400782afe33c299027cd" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 46865 } } }
I20260928 14:56:38.990310 3132 sys_catalog.cc:455] T 00000000000000000000000000000000 P d364c65f3538400782afe33c299027cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d364c65f3538400782afe33c299027cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d364c65f3538400782afe33c299027cd" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 46865 } } }
I20260928 14:56:38.990849 3132 sys_catalog.cc:458] T 00000000000000000000000000000000 P d364c65f3538400782afe33c299027cd [sys.catalog]: This master's current role is: LEADER
I20260928 14:56:38.990854 3133 sys_catalog.cc:458] T 00000000000000000000000000000000 P d364c65f3538400782afe33c299027cd [sys.catalog]: This master's current role is: LEADER
I20260928 14:56:38.993513 3139 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:56:38.999444 3139 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:56:39.009510 3139 catalog_manager.cc:1347] Generated new cluster ID: 9c43a34f24524040bc56dd705e44c76e
I20260928 14:56:39.009646 3139 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:56:39.035674 3139 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:56:39.037192 3139 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:56:39.053978 3139 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d364c65f3538400782afe33c299027cd: Generated new TSK 0
I20260928 14:56:39.055088 3139 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.40.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab.
Entry for principal kudu/127.31.40.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab.
I20260928 14:56:39.117645 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.1:0
--local_ip_for_outbound_sockets=127.31.40.1
--webserver_interface=127.31.40.1
--webserver_port=0
--tserver_master_addrs=127.31.40.62:46865
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab
--principal=kudu/127.31.40.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:38561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:56:39.466806 3152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:56:39.467231 3152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:56:39.467417 3152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:56:39.474946 3152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:56:39.475087 3152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:56:39.475288 3152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.1
I20260928 14:56:39.485796 3152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:38561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab
--principal=kudu/127.31.40.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.40.1
--webserver_port=0
--tserver_master_addrs=127.31.40.62:46865
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.1
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:56:39.487329 3152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:56:39.488813 3152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:56:39.504287 3157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:39.504503 3158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:39.504601 3160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:39.505180 3152 server_base.cc:1034] running on GCE node
I20260928 14:56:39.505841 3152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:56:39.507009 3152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:56:39.508267 3152 hybrid_clock.cc:648] HybridClock initialized: now 1790607399508228 us; error 40 us; skew 500 ppm
I20260928 14:56:39.508694 3152 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:56:39 dist-test-slave-j968 krb5kdc[3045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607399, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:56:39.512331 3152 init.cc:377] Logged in from keytab as kudu/127.31.40.1@KRBTEST.COM (short username kudu)
I20260928 14:56:39.514005 3152 webserver.cc:460] Webserver started at http://127.31.40.1:33845/ using document root <none> and password file <none>
I20260928 14:56:39.514707 3152 fs_manager.cc:359] Metadata directory not provided
I20260928 14:56:39.514813 3152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:56:39.515091 3152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:56:39.518239 3152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/ts-0/data/instance:
uuid: "f92fc127f3164d05b6fa12cb7b957e0d"
format_stamp: "Formatted at 2026-09-28 14:56:39 on dist-test-slave-j968"
server_key: "13d28a9ea7e15c9e69fc33d4a3fdc1c3"
server_key_iv: "5d7d4df4f653399a56de4a5917906723"
server_key_version: "encryptionkey@0"
I20260928 14:56:39.519011 3152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/ts-0/wal/instance:
uuid: "f92fc127f3164d05b6fa12cb7b957e0d"
format_stamp: "Formatted at 2026-09-28 14:56:39 on dist-test-slave-j968"
server_key: "13d28a9ea7e15c9e69fc33d4a3fdc1c3"
server_key_iv: "5d7d4df4f653399a56de4a5917906723"
server_key_version: "encryptionkey@0"
I20260928 14:56:39.524698 3152 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 14:56:39.528491 3167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:39.530156 3152 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 14:56:39.530311 3152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/ts-0/wal
uuid: "f92fc127f3164d05b6fa12cb7b957e0d"
format_stamp: "Formatted at 2026-09-28 14:56:39 on dist-test-slave-j968"
server_key: "13d28a9ea7e15c9e69fc33d4a3fdc1c3"
server_key_iv: "5d7d4df4f653399a56de4a5917906723"
server_key_version: "encryptionkey@0"
I20260928 14:56:39.530505 3152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:56:39.543569 3152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:56:39.548341 3152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:56:39.548599 3152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:56:39.549548 3152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:56:39.550869 3152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:56:39.550956 3152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:39.551040 3152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:56:39.551075 3152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:39.586525 3152 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.1:43545
I20260928 14:56:39.586575 3280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.1:43545 every 8 connection(s)
I20260928 14:56:39.588070 3152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
I20260928 14:56:39.593286 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 3152
I20260928 14:56:39.593550 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/ts-0/wal/instance
I20260928 14:56:39.594345 31904 external_mini_cluster.cc:1428] Setting key 39f8a0b48dcb76b443d619fe89d7ebe9
Sep 28 14:56:39 dist-test-slave-j968 krb5kdc[3045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607399, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.1@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:56:39.614880 3283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:39.591081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:52185 (local address 127.31.40.62:46865)
0928 14:56:39.591772 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:39.591777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:39.592707 (+ 930us) server_negotiation.cc:407] Connection header received
0928 14:56:39.595511 (+ 2804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:39.595516 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:39.595606 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:39.595738 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:39.597761 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:39.598585 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:39.601275 (+ 2690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:39.602080 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:39.602247 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:39.606953 (+ 4706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:39.606964 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:39.606983 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:39.607038 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:39.610986 (+ 3948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:39.611943 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:39.611952 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:39.611957 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:39.612144 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:39.612682 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:39.612690 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:39.612696 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:39.613156 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:39.613342 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:39.614182 (+ 840us) server_negotiation.cc:299] Negotiation successful
0928 14:56:39.614451 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":452,"threads_started":1}
I20260928 14:56:39.615701 3284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:39.591699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:46865 (local address 127.31.40.1:52185)
0928 14:56:39.592544 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:39.592585 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:39.595236 (+ 2651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:39.596017 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:39.596027 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:39.596680 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:56:39.597556 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:39.597585 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:39.598781 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:39.598789 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:39.601086 (+ 2297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:39.601112 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:39.602350 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:39.602360 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:39.602827 (+ 467us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:39.603760 (+ 933us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:56:39.603800 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:56:39.606636 (+ 2836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:56:39.611195 (+ 4559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:39.611204 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:39.611221 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:39.611743 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:39.612301 (+ 558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:39.612305 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:39.612308 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:39.612529 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:39.613324 (+ 795us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:56:39.613332 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:56:39.613732 (+ 400us) client_negotiation.cc:769] Sending connection context
0928 14:56:39.614007 (+ 275us) client_negotiation.cc:240] Negotiation successful
0928 14:56:39.614451 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":335,"threads_started":1}
WARNING: no policy specified for kudu/127.31.40.2@KRBTEST.COM; defaulting to no policy
I20260928 14:56:39.617549 3281 heartbeater.cc:347] Connected to a master server at 127.31.40.62:46865
I20260928 14:56:39.617908 3281 heartbeater.cc:464] Registering TS with master...
I20260928 14:56:39.618746 3281 heartbeater.cc:511] Master 127.31.40.62:46865 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.2@KRBTEST.COM" created.
I20260928 14:56:39.621313 3091 ts_manager.cc:194] Registered new tserver with Master: f92fc127f3164d05b6fa12cb7b957e0d (127.31.40.1:43545)
I20260928 14:56:39.622866 3091 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.1@KRBTEST.COM'} at 127.31.40.1:52185
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab.
Entry for principal kudu/127.31.40.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab.
I20260928 14:56:39.647133 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.2:0
--local_ip_for_outbound_sockets=127.31.40.2
--webserver_interface=127.31.40.2
--webserver_port=0
--tserver_master_addrs=127.31.40.62:46865
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab
--principal=kudu/127.31.40.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:38561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:56:39.997293 3287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:56:39.997608 3287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:56:39.997794 3287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:56:40.005584 3287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:56:40.005713 3287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:56:40.005891 3287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.2
I20260928 14:56:40.016085 3287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:38561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab
--principal=kudu/127.31.40.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.40.2
--webserver_port=0
--tserver_master_addrs=127.31.40.62:46865
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.2
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:56:40.017505 3287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:56:40.018893 3287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:56:40.032702 3293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:40.032699 3292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:40.032729 3295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:40.033721 3287 server_base.cc:1034] running on GCE node
I20260928 14:56:40.034474 3287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:56:40.035595 3287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:56:40.036803 3287 hybrid_clock.cc:648] HybridClock initialized: now 1790607400036709 us; error 82 us; skew 500 ppm
I20260928 14:56:40.037158 3287 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:56:40 dist-test-slave-j968 krb5kdc[3045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607400, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:56:40.040997 3287 init.cc:377] Logged in from keytab as kudu/127.31.40.2@KRBTEST.COM (short username kudu)
I20260928 14:56:40.042819 3287 webserver.cc:460] Webserver started at http://127.31.40.2:44185/ using document root <none> and password file <none>
I20260928 14:56:40.043656 3287 fs_manager.cc:359] Metadata directory not provided
I20260928 14:56:40.043828 3287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:56:40.044246 3287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:56:40.047628 3287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/ts-1/data/instance:
uuid: "93dd4a62a6084dae898e470f822fc9e2"
format_stamp: "Formatted at 2026-09-28 14:56:40 on dist-test-slave-j968"
server_key: "16046e56fc280ce2f35056b142d5a32f"
server_key_iv: "abd5dea899c9f65bc7d7edd6136ad55e"
server_key_version: "encryptionkey@0"
I20260928 14:56:40.048619 3287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/ts-1/wal/instance:
uuid: "93dd4a62a6084dae898e470f822fc9e2"
format_stamp: "Formatted at 2026-09-28 14:56:40 on dist-test-slave-j968"
server_key: "16046e56fc280ce2f35056b142d5a32f"
server_key_iv: "abd5dea899c9f65bc7d7edd6136ad55e"
server_key_version: "encryptionkey@0"
I20260928 14:56:40.054634 3287 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 14:56:40.058606 3302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:40.060504 3287 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 14:56:40.060730 3287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/ts-1/wal
uuid: "93dd4a62a6084dae898e470f822fc9e2"
format_stamp: "Formatted at 2026-09-28 14:56:40 on dist-test-slave-j968"
server_key: "16046e56fc280ce2f35056b142d5a32f"
server_key_iv: "abd5dea899c9f65bc7d7edd6136ad55e"
server_key_version: "encryptionkey@0"
I20260928 14:56:40.060977 3287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:56:40.087953 3287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:56:40.093024 3287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:56:40.093361 3287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:56:40.094426 3287 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:56:40.095834 3287 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:56:40.095961 3287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:40.096115 3287 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:56:40.096189 3287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:40.132385 3287 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.2:44069
I20260928 14:56:40.132503 3415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.2:44069 every 8 connection(s)
I20260928 14:56:40.134459 3287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
I20260928 14:56:40.139902 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 3287
I20260928 14:56:40.140115 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/ts-1/wal/instance
I20260928 14:56:40.140730 31904 external_mini_cluster.cc:1428] Setting key 3c2e447cd60226c8d97a7c9b68ff8905
Sep 28 14:56:40 dist-test-slave-j968 krb5kdc[3045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607400, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.2@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:56:40.160038 3418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:40.138282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:43085 (local address 127.31.40.62:46865)
0928 14:56:40.138976 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:40.139017 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:40.140644 (+ 1627us) server_negotiation.cc:407] Connection header received
0928 14:56:40.143180 (+ 2536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:40.143186 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:40.143275 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:40.143426 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:40.145293 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:40.146096 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:40.147895 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:40.148386 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:40.148545 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:40.153309 (+ 4764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:40.153314 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:40.153317 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:40.153355 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:40.156162 (+ 2807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:40.156978 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:40.156983 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:40.156985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:40.157122 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:40.157672 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:40.157681 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:40.157686 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:40.158186 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:40.158359 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:40.159155 (+ 796us) server_negotiation.cc:299] Negotiation successful
0928 14:56:40.159425 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":309,"threads_started":1}
I20260928 14:56:40.160710 3419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:40.138944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:46865 (local address 127.31.40.2:43085)
0928 14:56:40.140063 (+ 1119us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:40.140133 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:40.142859 (+ 2726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:40.143711 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:40.143721 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:40.144403 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:56:40.145095 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:40.145115 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:40.146270 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:40.146279 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:40.147680 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:40.147694 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:40.148657 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:40.148664 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:40.149014 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:40.149810 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:56:40.149851 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:56:40.152953 (+ 3102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:56:40.156331 (+ 3378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:40.156341 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:40.156365 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:40.156829 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:40.157237 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:40.157242 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:40.157246 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:40.157483 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:40.158353 (+ 870us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:56:40.158359 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:56:40.158733 (+ 374us) client_negotiation.cc:769] Sending connection context
0928 14:56:40.158968 (+ 235us) client_negotiation.cc:240] Negotiation successful
0928 14:56:40.159446 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"thread_start_us":436,"threads_started":1}
WARNING: no policy specified for kudu/127.31.40.3@KRBTEST.COM; defaulting to no policy
I20260928 14:56:40.162422 3416 heartbeater.cc:347] Connected to a master server at 127.31.40.62:46865
I20260928 14:56:40.162786 3416 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.3@KRBTEST.COM" created.
I20260928 14:56:40.163738 3416 heartbeater.cc:511] Master 127.31.40.62:46865 requested a full tablet report, sending...
I20260928 14:56:40.165434 3091 ts_manager.cc:194] Registered new tserver with Master: 93dd4a62a6084dae898e470f822fc9e2 (127.31.40.2:44069)
I20260928 14:56:40.166404 3091 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.2@KRBTEST.COM'} at 127.31.40.2:43085
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab.
Entry for principal kudu/127.31.40.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab.
I20260928 14:56:40.187791 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.3:0
--local_ip_for_outbound_sockets=127.31.40.3
--webserver_interface=127.31.40.3
--webserver_port=0
--tserver_master_addrs=127.31.40.62:46865
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab
--principal=kudu/127.31.40.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:38561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:56:40.525445 3422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:56:40.525837 3422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:56:40.526072 3422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:56:40.534096 3422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:56:40.534238 3422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:56:40.534415 3422 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.3
I20260928 14:56:40.544792 3422 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:38561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab
--principal=kudu/127.31.40.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.40.3
--webserver_port=0
--tserver_master_addrs=127.31.40.62:46865
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.3
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:56:40.546319 3422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:56:40.547782 3422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:56:40.562070 3428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:40.562131 3427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:40.562045 3430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:40.562508 3422 server_base.cc:1034] running on GCE node
I20260928 14:56:40.563395 3422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:56:40.564491 3422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:56:40.565707 3422 hybrid_clock.cc:648] HybridClock initialized: now 1790607400565621 us; error 64 us; skew 500 ppm
I20260928 14:56:40.566134 3422 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:56:40 dist-test-slave-j968 krb5kdc[3045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607400, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:56:40.569443 3422 init.cc:377] Logged in from keytab as kudu/127.31.40.3@KRBTEST.COM (short username kudu)
I20260928 14:56:40.570955 3422 webserver.cc:460] Webserver started at http://127.31.40.3:33015/ using document root <none> and password file <none>
I20260928 14:56:40.571720 3422 fs_manager.cc:359] Metadata directory not provided
I20260928 14:56:40.571916 3422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:56:40.572271 3422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:56:40.575354 3422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/ts-2/data/instance:
uuid: "adc4ad09dc034076a17e048875d707f6"
format_stamp: "Formatted at 2026-09-28 14:56:40 on dist-test-slave-j968"
server_key: "d685fdb7f801e423736681124f060b12"
server_key_iv: "a890a97eb509e21ce92348bd34db18c3"
server_key_version: "encryptionkey@0"
I20260928 14:56:40.576274 3422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/ts-2/wal/instance:
uuid: "adc4ad09dc034076a17e048875d707f6"
format_stamp: "Formatted at 2026-09-28 14:56:40 on dist-test-slave-j968"
server_key: "d685fdb7f801e423736681124f060b12"
server_key_iv: "a890a97eb509e21ce92348bd34db18c3"
server_key_version: "encryptionkey@0"
I20260928 14:56:40.581701 3422 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 14:56:40.585458 3437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:40.587088 3422 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 14:56:40.587267 3422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/ts-2/wal
uuid: "adc4ad09dc034076a17e048875d707f6"
format_stamp: "Formatted at 2026-09-28 14:56:40 on dist-test-slave-j968"
server_key: "d685fdb7f801e423736681124f060b12"
server_key_iv: "a890a97eb509e21ce92348bd34db18c3"
server_key_version: "encryptionkey@0"
I20260928 14:56:40.587509 3422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:56:40.604871 3422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:56:40.609589 3422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:56:40.609930 3422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:56:40.611025 3422 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:56:40.612597 3422 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:56:40.612718 3422 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:40.612890 3422 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:56:40.612970 3422 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:40.627036 3281 heartbeater.cc:503] Master 127.31.40.62:46865 was elected leader, sending a full tablet report...
I20260928 14:56:40.652609 3422 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.3:38251
I20260928 14:56:40.652655 3549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.3:38251 every 8 connection(s)
I20260928 14:56:40.654076 3422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
I20260928 14:56:40.658605 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 3422
I20260928 14:56:40.658810 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607365362981-31904-0/minicluster-data/ts-2/wal/instance
I20260928 14:56:40.659356 31904 external_mini_cluster.cc:1428] Setting key fcafd79dd22bce09594cab38652c2138
Sep 28 14:56:40 dist-test-slave-j968 krb5kdc[3045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607400, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.3@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:56:40.675312 3418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:40.656884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:46707 (local address 127.31.40.62:46865)
0928 14:56:40.657085 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:40.657089 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:40.658358 (+ 1269us) server_negotiation.cc:407] Connection header received
0928 14:56:40.660868 (+ 2510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:40.660873 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:40.661045 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:40.661136 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:40.663059 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:40.663780 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:40.665497 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:40.665897 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:40.666003 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:40.669432 (+ 3429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:40.669437 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:40.669441 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:40.669473 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:40.672218 (+ 2745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:40.672891 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:40.672895 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:40.672898 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:40.673030 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:40.673490 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:40.673495 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:40.673498 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:40.673786 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:40.673979 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:40.674743 (+ 764us) server_negotiation.cc:299] Negotiation successful
0928 14:56:40.674973 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 14:56:40.676401 3552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:40.657347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:46865 (local address 127.31.40.3:46707)
0928 14:56:40.658101 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:40.658200 (+ 99us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:40.660543 (+ 2343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:40.661454 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:40.661466 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:40.662116 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:56:40.662903 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:40.662922 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:40.664062 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:40.664066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:40.665346 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:40.665358 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:40.666089 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:40.666094 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:40.666294 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:40.666930 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:56:40.666955 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:56:40.669234 (+ 2279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:56:40.672342 (+ 3108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:40.672349 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:40.672363 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:40.672770 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:40.673138 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:40.673143 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:40.673146 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:40.673371 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:40.673924 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:56:40.673932 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:56:40.674326 (+ 394us) client_negotiation.cc:769] Sending connection context
0928 14:56:40.674641 (+ 315us) client_negotiation.cc:240] Negotiation successful
0928 14:56:40.675186 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":294,"threads_started":1}
I20260928 14:56:40.677940 3550 heartbeater.cc:347] Connected to a master server at 127.31.40.62:46865
I20260928 14:56:40.678284 3550 heartbeater.cc:464] Registering TS with master...
I20260928 14:56:40.679059 3550 heartbeater.cc:511] Master 127.31.40.62:46865 requested a full tablet report, sending...
I20260928 14:56:40.680667 3091 ts_manager.cc:194] Registered new tserver with Master: adc4ad09dc034076a17e048875d707f6 (127.31.40.3:38251)
I20260928 14:56:40.681567 3091 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.3@KRBTEST.COM'} at 127.31.40.3:46707
I20260928 14:56:40.687613 31904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 14:56:40 dist-test-slave-j968 krb5kdc[3045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 14:56:40 dist-test-slave-j968 krb5kdc[3045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.1@KRBTEST.COM
I20260928 14:56:40.742810 3557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:40.725733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46626 (local address 127.31.40.1:43545)
0928 14:56:40.726521 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:40.726529 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:40.726559 (+ 30us) server_negotiation.cc:407] Connection header received
0928 14:56:40.726697 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:40.726705 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:40.726949 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:40.727149 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:40.728698 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:40.729773 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:40.731031 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:40.731613 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:40.731721 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:40.736055 (+ 4334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:40.736073 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:40.736091 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:40.736149 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:40.739499 (+ 3350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:40.740465 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:40.740474 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:40.740477 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:40.740647 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:40.741076 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:40.741080 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:40.741082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:40.741482 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:40.741730 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:40.742137 (+ 407us) server_negotiation.cc:299] Negotiation successful
0928 14:56:40.742441 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":469,"threads_started":1}
Sep 28 14:56:40 dist-test-slave-j968 krb5kdc[3045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:56:40.763129 3418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:40.748281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44712 (local address 127.31.40.62:46865)
0928 14:56:40.748534 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:40.748541 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:40.748677 (+ 136us) server_negotiation.cc:407] Connection header received
0928 14:56:40.748844 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:40.748848 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:40.748920 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:40.749069 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:40.750398 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:40.751248 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:40.752649 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:40.753147 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:40.753252 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:40.757280 (+ 4028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:40.757285 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:40.757289 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:40.757329 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:40.760111 (+ 2782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:40.760836 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:40.760841 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:40.760843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:40.760996 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:40.761394 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:40.761399 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:40.761401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:40.761761 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:40.762074 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:40.762415 (+ 341us) server_negotiation.cc:299] Negotiation successful
0928 14:56:40.762630 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260928 14:56:40.763904 3091 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44712
I20260928 14:56:40.765141 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 3152
I20260928 14:56:40.778625 3557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:40.766168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46628 (local address 127.31.40.1:43545)
0928 14:56:40.766342 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:40.766346 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:40.766565 (+ 219us) server_negotiation.cc:407] Connection header received
0928 14:56:40.766648 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:40.766653 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:40.766725 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:40.766815 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:40.768125 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:40.768869 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:40.770203 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:40.770655 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:40.770761 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:40.773201 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:40.773206 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:40.773210 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:40.773245 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:40.776108 (+ 2863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:40.776865 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:40.776870 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:40.776872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:40.777009 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:40.777390 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:40.777393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:40.777395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:40.777657 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:40.777832 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:40.778197 (+ 365us) server_negotiation.cc:299] Negotiation successful
0928 14:56:40.778368 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 14:56:40.779332 3276 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:56:40.863855 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 3152
I20260928 14:56:40.881090 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 3287
Sep 28 14:56:40 dist-test-slave-j968 krb5kdc[3045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.2@KRBTEST.COM
I20260928 14:56:40.899291 3562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:40.882136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56292 (local address 127.31.40.2:44069)
0928 14:56:40.882690 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:40.882697 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:40.882722 (+ 25us) server_negotiation.cc:407] Connection header received
0928 14:56:40.882782 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:40.882788 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:40.883007 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:40.883186 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:40.884556 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:40.885619 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:40.886917 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:40.887577 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:40.887732 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:40.892639 (+ 4907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:40.892654 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:40.892672 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:40.892774 (+ 102us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:40.895954 (+ 3180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:40.896772 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:40.896786 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:40.896794 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:40.896946 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:40.897408 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:40.897414 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:40.897418 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:40.897838 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:40.898196 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:40.898568 (+ 372us) server_negotiation.cc:299] Negotiation successful
0928 14:56:40.898883 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":274,"threads_started":1}
I20260928 14:56:40.900883 3411 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:56:40.986321 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 3287
I20260928 14:56:41.004400 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 3422
Sep 28 14:56:41 dist-test-slave-j968 krb5kdc[3045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.3@KRBTEST.COM
I20260928 14:56:41.022799 3564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:41.005629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49584 (local address 127.31.40.3:38251)
0928 14:56:41.006352 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:41.006360 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:41.006392 (+ 32us) server_negotiation.cc:407] Connection header received
0928 14:56:41.006459 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:41.006465 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:41.006693 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:41.006840 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:41.008217 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:41.009558 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:41.011020 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:41.011642 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:41.011760 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:41.016650 (+ 4890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:41.016665 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:41.016682 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:41.016725 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:41.019784 (+ 3059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:41.020708 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:41.020716 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:41.020718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:41.020859 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:41.021255 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:41.021258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:41.021260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:41.021634 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:41.021886 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:41.022259 (+ 373us) server_negotiation.cc:299] Negotiation successful
0928 14:56:41.022466 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":378,"threads_started":1}
I20260928 14:56:41.024206 3545 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:56:41.111186 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 3422
I20260928 14:56:41.128213 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 3059
I20260928 14:56:41.129537 3121 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:56:41.235432 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 3059
2026-09-28T14:56:41Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4077 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:56:41 dist-test-slave-j968 krb5kdc[3570](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:56:41 dist-test-slave-j968 krb5kdc[3570](info): set up 2 sockets
Sep 28 14:56:41 dist-test-slave-j968 krb5kdc[3570](info): commencing operation
krb5kdc: starting...
W20260928 14:56:42.415076 31904 mini_kdc.cc:121] Time spent starting KDC: real 1.136s user 0.002s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Sep 28 14:56:42 dist-test-slave-j968 krb5kdc[3570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:56:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:56:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab.
Entry for principal kudu/127.31.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab.
I20260928 14:56:42.645016 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:34795
--webserver_interface=127.31.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:45101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.40.62:34795
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:56:42.992765 3583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:56:42.993103 3583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:56:42.993278 3583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:56:43.000921 3583 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:56:43.001056 3583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:56:43.001140 3583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:56:43.001235 3583 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:56:43.001323 3583 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:56:43.012046 3583 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:45101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.40.62:34795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:34795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:56:43.013998 3583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:56:43.015647 3583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:56:43.028424 3591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:43.028435 3589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:43.028529 3588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:43.030615 3583 server_base.cc:1034] running on GCE node
I20260928 14:56:43.031705 3583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:56:43.033365 3583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:56:43.034660 3583 hybrid_clock.cc:648] HybridClock initialized: now 1790607403034619 us; error 36 us; skew 500 ppm
I20260928 14:56:43.035072 3583 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:56:43 dist-test-slave-j968 krb5kdc[3570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607403, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:56:43.038739 3583 init.cc:377] Logged in from keytab as kudu/127.31.40.62@KRBTEST.COM (short username kudu)
I20260928 14:56:43.040232 3583 fs_manager.cc:359] Metadata directory not provided
I20260928 14:56:43.040374 3583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:56:43.040740 3583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:56:43.044057 3583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/master-0/data/instance:
uuid: "787b318e05c44c40a5f16295168835e6"
format_stamp: "Formatted at 2026-09-28 14:56:43 on dist-test-slave-j968"
server_key: "cfa07b87ecce62a8aaa7b2f89f1b1057"
server_key_iv: "289d781e603aef12acf4e036c14543b7"
server_key_version: "encryptionkey@0"
I20260928 14:56:43.044925 3583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/master-0/wal/instance:
uuid: "787b318e05c44c40a5f16295168835e6"
format_stamp: "Formatted at 2026-09-28 14:56:43 on dist-test-slave-j968"
server_key: "cfa07b87ecce62a8aaa7b2f89f1b1057"
server_key_iv: "289d781e603aef12acf4e036c14543b7"
server_key_version: "encryptionkey@0"
I20260928 14:56:43.050633 3583 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260928 14:56:43.054811 3597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:43.056686 3583 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 14:56:43.056949 3583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/master-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/master-0/wal
uuid: "787b318e05c44c40a5f16295168835e6"
format_stamp: "Formatted at 2026-09-28 14:56:43 on dist-test-slave-j968"
server_key: "cfa07b87ecce62a8aaa7b2f89f1b1057"
server_key_iv: "289d781e603aef12acf4e036c14543b7"
server_key_version: "encryptionkey@0"
I20260928 14:56:43.057245 3583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:56:43.076069 3583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:56:43.081602 3583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:56:43.081966 3583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:56:43.103456 3583 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.62:34795
I20260928 14:56:43.103480 3649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.62:34795 every 8 connection(s)
I20260928 14:56:43.104686 3583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
I20260928 14:56:43.107383 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 3583
I20260928 14:56:43.107661 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/master-0/wal/instance
I20260928 14:56:43.108475 31904 external_mini_cluster.cc:1428] Setting key e58a51adc6e44882808d98d2b5313a7d
I20260928 14:56:43.109423 3650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 14:56:43 dist-test-slave-j968 krb5kdc[3570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:56:43.126092 3650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 787b318e05c44c40a5f16295168835e6: Bootstrap starting.
I20260928 14:56:43.131335 3650 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 787b318e05c44c40a5f16295168835e6: Neither blocks nor log segments found. Creating new log.
I20260928 14:56:43.133176 3650 log.cc:824] T 00000000000000000000000000000000 P 787b318e05c44c40a5f16295168835e6: Log is configured to *not* fsync() on all Append() calls
I20260928 14:56:43.138645 3650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 787b318e05c44c40a5f16295168835e6: No bootstrap required, opened a new log
I20260928 14:56:43.139600 3653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:43.110819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60452 (local address 127.31.40.62:34795)
0928 14:56:43.111786 (+ 967us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:43.111805 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:43.111887 (+ 82us) server_negotiation.cc:407] Connection header received
0928 14:56:43.113787 (+ 1900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:43.113820 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:43.114462 (+ 642us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:43.115007 (+ 545us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:43.116677 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:43.119037 (+ 2360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:43.120638 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:43.121887 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:43.122147 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:43.126432 (+ 4285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:43.126447 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:43.126489 (+ 42us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:43.126549 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:43.130945 (+ 4396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:43.132195 (+ 1250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:43.132206 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:43.132216 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:43.132497 (+ 281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:43.133121 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:43.133127 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:43.133131 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:43.134447 (+ 1316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:43.135169 (+ 722us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:43.136925 (+ 1756us) server_negotiation.cc:299] Negotiation successful
0928 14:56:43.137497 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":681,"thread_start_us":401,"threads_started":1}
I20260928 14:56:43.153047 3650 raft_consensus.cc:348] T 00000000000000000000000000000000 P 787b318e05c44c40a5f16295168835e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "787b318e05c44c40a5f16295168835e6" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34795 } }
I20260928 14:56:43.153791 3650 raft_consensus.cc:374] T 00000000000000000000000000000000 P 787b318e05c44c40a5f16295168835e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:56:43.153942 3650 raft_consensus.cc:729] T 00000000000000000000000000000000 P 787b318e05c44c40a5f16295168835e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 787b318e05c44c40a5f16295168835e6, State: Initialized, Role: FOLLOWER
I20260928 14:56:43.154680 3650 consensus_queue.cc:260] T 00000000000000000000000000000000 P 787b318e05c44c40a5f16295168835e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "787b318e05c44c40a5f16295168835e6" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34795 } }
I20260928 14:56:43.155021 3650 raft_consensus.cc:388] T 00000000000000000000000000000000 P 787b318e05c44c40a5f16295168835e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:56:43.155200 3650 raft_consensus.cc:482] T 00000000000000000000000000000000 P 787b318e05c44c40a5f16295168835e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:56:43.155417 3650 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 787b318e05c44c40a5f16295168835e6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:56:43.159334 3650 raft_consensus.cc:504] T 00000000000000000000000000000000 P 787b318e05c44c40a5f16295168835e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "787b318e05c44c40a5f16295168835e6" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34795 } }
I20260928 14:56:43.160218 3650 leader_election.cc:302] T 00000000000000000000000000000000 P 787b318e05c44c40a5f16295168835e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 787b318e05c44c40a5f16295168835e6; no voters:
I20260928 14:56:43.161072 3650 leader_election.cc:288] T 00000000000000000000000000000000 P 787b318e05c44c40a5f16295168835e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:56:43.161231 3655 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 787b318e05c44c40a5f16295168835e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:56:43.162395 3655 raft_consensus.cc:686] T 00000000000000000000000000000000 P 787b318e05c44c40a5f16295168835e6 [term 1 LEADER]: Becoming Leader. State: Replica: 787b318e05c44c40a5f16295168835e6, State: Running, Role: LEADER
I20260928 14:56:43.163225 3655 consensus_queue.cc:237] T 00000000000000000000000000000000 P 787b318e05c44c40a5f16295168835e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "787b318e05c44c40a5f16295168835e6" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34795 } }
I20260928 14:56:43.164309 3650 sys_catalog.cc:564] T 00000000000000000000000000000000 P 787b318e05c44c40a5f16295168835e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:56:43.167167 3657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 787b318e05c44c40a5f16295168835e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 787b318e05c44c40a5f16295168835e6. Latest consensus state: current_term: 1 leader_uuid: "787b318e05c44c40a5f16295168835e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "787b318e05c44c40a5f16295168835e6" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34795 } } }
I20260928 14:56:43.167732 3657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 787b318e05c44c40a5f16295168835e6 [sys.catalog]: This master's current role is: LEADER
I20260928 14:56:43.169225 3656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 787b318e05c44c40a5f16295168835e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "787b318e05c44c40a5f16295168835e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "787b318e05c44c40a5f16295168835e6" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34795 } } }
I20260928 14:56:43.169787 3656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 787b318e05c44c40a5f16295168835e6 [sys.catalog]: This master's current role is: LEADER
I20260928 14:56:43.172996 3661 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:56:43.178272 3661 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:56:43.189333 3661 catalog_manager.cc:1347] Generated new cluster ID: 9945a5070ae147768b72bff1d83860dc
I20260928 14:56:43.189596 3661 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:56:43.206554 3661 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:56:43.208158 3661 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:56:43.216632 3661 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 787b318e05c44c40a5f16295168835e6: Generated new TSK 0
I20260928 14:56:43.217929 3661 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.40.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab.
Entry for principal kudu/127.31.40.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab.
I20260928 14:56:43.277822 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.1:0
--local_ip_for_outbound_sockets=127.31.40.1
--webserver_interface=127.31.40.1
--webserver_port=0
--tserver_master_addrs=127.31.40.62:34795
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab
--principal=kudu/127.31.40.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:45101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:56:43.626916 3676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:56:43.627295 3676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:56:43.627468 3676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:56:43.635021 3676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:56:43.635218 3676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:56:43.635402 3676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.1
I20260928 14:56:43.646306 3676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:45101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab
--principal=kudu/127.31.40.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.40.1
--webserver_port=0
--tserver_master_addrs=127.31.40.62:34795
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.1
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:56:43.647747 3676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:56:43.649163 3676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:56:43.663272 3684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:43.663340 3682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:43.663616 3681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:43.664613 3676 server_base.cc:1034] running on GCE node
I20260928 14:56:43.665307 3676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:56:43.666494 3676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:56:43.667749 3676 hybrid_clock.cc:648] HybridClock initialized: now 1790607403667724 us; error 62 us; skew 500 ppm
I20260928 14:56:43.668179 3676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:56:43 dist-test-slave-j968 krb5kdc[3570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607403, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:56:43.671612 3676 init.cc:377] Logged in from keytab as kudu/127.31.40.1@KRBTEST.COM (short username kudu)
I20260928 14:56:43.673105 3676 fs_manager.cc:359] Metadata directory not provided
I20260928 14:56:43.673245 3676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:56:43.673578 3676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:56:43.676713 3676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/ts-0/data/instance:
uuid: "c224246c15ff4c8587a6342fb5d3973f"
format_stamp: "Formatted at 2026-09-28 14:56:43 on dist-test-slave-j968"
server_key: "496bed0a957c795e7d3c83cff2992c39"
server_key_iv: "41ae3e7d13a4ac4085d7ecb681b3b7eb"
server_key_version: "encryptionkey@0"
I20260928 14:56:43.677552 3676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/ts-0/wal/instance:
uuid: "c224246c15ff4c8587a6342fb5d3973f"
format_stamp: "Formatted at 2026-09-28 14:56:43 on dist-test-slave-j968"
server_key: "496bed0a957c795e7d3c83cff2992c39"
server_key_iv: "41ae3e7d13a4ac4085d7ecb681b3b7eb"
server_key_version: "encryptionkey@0"
I20260928 14:56:43.682904 3676 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 14:56:43.686582 3690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:43.688104 3676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:56:43.688283 3676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/ts-0/wal
uuid: "c224246c15ff4c8587a6342fb5d3973f"
format_stamp: "Formatted at 2026-09-28 14:56:43 on dist-test-slave-j968"
server_key: "496bed0a957c795e7d3c83cff2992c39"
server_key_iv: "41ae3e7d13a4ac4085d7ecb681b3b7eb"
server_key_version: "encryptionkey@0"
I20260928 14:56:43.688506 3676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:56:43.710199 3676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:56:43.714612 3676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:56:43.714910 3676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:56:43.716218 3676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:56:43.717792 3676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:56:43.717918 3676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:43.718060 3676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:56:43.718153 3676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:43.752179 3676 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.1:42173
I20260928 14:56:43.752269 3802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.1:42173 every 8 connection(s)
I20260928 14:56:43.753098 3676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
I20260928 14:56:43.760957 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 3676
I20260928 14:56:43.761317 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/ts-0/wal/instance
I20260928 14:56:43.762120 31904 external_mini_cluster.cc:1428] Setting key 6341c720bf5653745716a9e5d8b30613
Sep 28 14:56:43 dist-test-slave-j968 krb5kdc[3570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607403, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.1@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:56:43.775350 3805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:43.755822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:53719 (local address 127.31.40.62:34795)
0928 14:56:43.756323 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:43.756327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:43.757402 (+ 1075us) server_negotiation.cc:407] Connection header received
0928 14:56:43.759296 (+ 1894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:43.759302 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:43.759393 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:43.759508 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:43.761238 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:43.762012 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:43.763838 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:43.764434 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:43.764576 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:43.768696 (+ 4120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:43.768702 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:43.768705 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:43.768741 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:43.772114 (+ 3373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:43.772948 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:43.772953 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:43.772955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:43.773080 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:43.773530 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:43.773539 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:43.773541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:43.773808 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:43.773964 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:43.774729 (+ 765us) server_negotiation.cc:299] Negotiation successful
0928 14:56:43.774963 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":249,"threads_started":1}
I20260928 14:56:43.776533 3806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:43.756474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34795 (local address 127.31.40.1:53719)
0928 14:56:43.757209 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:43.757256 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:43.759006 (+ 1750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:43.759759 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:43.759768 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:43.760390 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:56:43.761004 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:43.761022 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:43.762176 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:43.762180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:43.763677 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:43.763689 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:43.764868 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:43.764873 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:43.765084 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:43.766046 (+ 962us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:56:43.766071 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:56:43.768427 (+ 2356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:56:43.772283 (+ 3856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:43.772294 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:43.772314 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:43.772796 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:43.773195 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:43.773200 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:43.773204 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:43.773396 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:43.773954 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:56:43.773962 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:56:43.774328 (+ 366us) client_negotiation.cc:769] Sending connection context
0928 14:56:43.774619 (+ 291us) client_negotiation.cc:240] Negotiation successful
0928 14:56:43.775221 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":294,"threads_started":1}
I20260928 14:56:43.778684 3803 heartbeater.cc:347] Connected to a master server at 127.31.40.62:34795
I20260928 14:56:43.779059 3803 heartbeater.cc:464] Registering TS with master...
I20260928 14:56:43.779912 3803 heartbeater.cc:511] Master 127.31.40.62:34795 requested a full tablet report, sending...
I20260928 14:56:43.782647 3614 ts_manager.cc:194] Registered new tserver with Master: c224246c15ff4c8587a6342fb5d3973f (127.31.40.1:42173)
WARNING: no policy specified for kudu/127.31.40.2@KRBTEST.COM; defaulting to no policy
I20260928 14:56:43.784451 3614 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.1@KRBTEST.COM'} at 127.31.40.1:53719
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab.
Entry for principal kudu/127.31.40.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab.
I20260928 14:56:43.812230 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.2:0
--local_ip_for_outbound_sockets=127.31.40.2
--webserver_interface=127.31.40.2
--webserver_port=0
--tserver_master_addrs=127.31.40.62:34795
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab
--principal=kudu/127.31.40.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:45101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:56:44.154754 3809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:56:44.155110 3809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:56:44.155282 3809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:56:44.162992 3809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:56:44.163144 3809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:56:44.163317 3809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.2
I20260928 14:56:44.173461 3809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:45101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab
--principal=kudu/127.31.40.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.40.2
--webserver_port=0
--tserver_master_addrs=127.31.40.62:34795
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.2
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:56:44.174852 3809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:56:44.176288 3809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:56:44.190912 3814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:44.191047 3817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:44.191041 3815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:44.197201 3809 server_base.cc:1034] running on GCE node
I20260928 14:56:44.197971 3809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:56:44.199047 3809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:56:44.200263 3809 hybrid_clock.cc:648] HybridClock initialized: now 1790607404200207 us; error 46 us; skew 500 ppm
I20260928 14:56:44.200623 3809 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:56:44 dist-test-slave-j968 krb5kdc[3570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607404, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:56:44.210302 3809 init.cc:377] Logged in from keytab as kudu/127.31.40.2@KRBTEST.COM (short username kudu)
I20260928 14:56:44.211942 3809 fs_manager.cc:359] Metadata directory not provided
I20260928 14:56:44.212126 3809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:56:44.212527 3809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:56:44.216234 3809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/ts-1/data/instance:
uuid: "51ac507dc8c4475aaf5982433257c582"
format_stamp: "Formatted at 2026-09-28 14:56:44 on dist-test-slave-j968"
server_key: "ad314a112572fa5ed7b9c747a3db02d1"
server_key_iv: "d2352656fca4b317be8264952b268f10"
server_key_version: "encryptionkey@0"
I20260928 14:56:44.217378 3809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/ts-1/wal/instance:
uuid: "51ac507dc8c4475aaf5982433257c582"
format_stamp: "Formatted at 2026-09-28 14:56:44 on dist-test-slave-j968"
server_key: "ad314a112572fa5ed7b9c747a3db02d1"
server_key_iv: "d2352656fca4b317be8264952b268f10"
server_key_version: "encryptionkey@0"
I20260928 14:56:44.223639 3809 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260928 14:56:44.227984 3823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:44.229637 3809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:56:44.229828 3809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/ts-1/wal
uuid: "51ac507dc8c4475aaf5982433257c582"
format_stamp: "Formatted at 2026-09-28 14:56:44 on dist-test-slave-j968"
server_key: "ad314a112572fa5ed7b9c747a3db02d1"
server_key_iv: "d2352656fca4b317be8264952b268f10"
server_key_version: "encryptionkey@0"
I20260928 14:56:44.230110 3809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:56:44.248878 3809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:56:44.253494 3809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:56:44.253778 3809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:56:44.254945 3809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:56:44.256389 3809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:56:44.256510 3809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:44.256644 3809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:56:44.256729 3809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:44.293612 3809 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.2:37559
I20260928 14:56:44.293726 3937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.2:37559 every 8 connection(s)
I20260928 14:56:44.294860 3809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
I20260928 14:56:44.304270 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 3809
I20260928 14:56:44.304476 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/ts-1/wal/instance
I20260928 14:56:44.305112 31904 external_mini_cluster.cc:1428] Setting key 871b603b0f58d074fd93ed6d89f128fb
Sep 28 14:56:44 dist-test-slave-j968 krb5kdc[3570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607404, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.2@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:56:44.328822 3941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:44.303364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:50701 (local address 127.31.40.62:34795)
0928 14:56:44.303821 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:44.303828 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:44.303851 (+ 23us) server_negotiation.cc:407] Connection header received
0928 14:56:44.305303 (+ 1452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:44.307257 (+ 1954us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:44.307363 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:44.307505 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:44.309984 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:44.311032 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:44.313784 (+ 2752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:44.314614 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:44.314771 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:44.319646 (+ 4875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:44.319652 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:44.319655 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:44.319693 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:44.325261 (+ 5568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:44.326138 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:44.326144 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:44.326148 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:44.326290 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:44.326766 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:44.326771 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:44.326775 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:44.327152 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:44.327335 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:44.328176 (+ 841us) server_negotiation.cc:299] Negotiation successful
0928 14:56:44.328420 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":276,"threads_started":1}
I20260928 14:56:44.330041 3940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:44.301220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34795 (local address 127.31.40.2:50701)
0928 14:56:44.302223 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:44.302284 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:44.304784 (+ 2500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:44.307808 (+ 3024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:44.307822 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:44.308768 (+ 946us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:56:44.309742 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:44.309780 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:44.311191 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:44.311200 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:44.313577 (+ 2377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:44.313597 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:44.314874 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:44.314879 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:44.315098 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:44.315824 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:56:44.315847 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:56:44.319329 (+ 3482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:56:44.325452 (+ 6123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:44.325463 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:44.325495 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:44.325988 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:44.326417 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:44.326423 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:44.326426 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:44.326638 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:44.327310 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:56:44.327320 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:56:44.327725 (+ 405us) client_negotiation.cc:769] Sending connection context
0928 14:56:44.328068 (+ 343us) client_negotiation.cc:240] Negotiation successful
0928 14:56:44.328686 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":376,"threads_started":1}
I20260928 14:56:44.331740 3938 heartbeater.cc:347] Connected to a master server at 127.31.40.62:34795
I20260928 14:56:44.332144 3938 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.31.40.3@KRBTEST.COM; defaulting to no policy
I20260928 14:56:44.333097 3938 heartbeater.cc:511] Master 127.31.40.62:34795 requested a full tablet report, sending...
I20260928 14:56:44.335047 3614 ts_manager.cc:194] Registered new tserver with Master: 51ac507dc8c4475aaf5982433257c582 (127.31.40.2:37559)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.3@KRBTEST.COM" created.
I20260928 14:56:44.336460 3614 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.2@KRBTEST.COM'} at 127.31.40.2:50701
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab.
Entry for principal kudu/127.31.40.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab.
I20260928 14:56:44.364669 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.3:0
--local_ip_for_outbound_sockets=127.31.40.3
--webserver_interface=127.31.40.3
--webserver_port=0
--tserver_master_addrs=127.31.40.62:34795
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab
--principal=kudu/127.31.40.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:45101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:56:44.716251 3944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:56:44.716621 3944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:56:44.716805 3944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:56:44.724493 3944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:56:44.724694 3944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:56:44.724902 3944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.3
I20260928 14:56:44.735283 3944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:45101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab
--principal=kudu/127.31.40.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.40.3
--webserver_port=0
--tserver_master_addrs=127.31.40.62:34795
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.3
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:56:44.736797 3944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:56:44.738265 3944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:56:44.752866 3949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:44.752887 3950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:44.753737 3952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:44.754328 3944 server_base.cc:1034] running on GCE node
I20260928 14:56:44.755089 3944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:56:44.756455 3944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:56:44.757722 3944 hybrid_clock.cc:648] HybridClock initialized: now 1790607404757641 us; error 76 us; skew 500 ppm
I20260928 14:56:44.758119 3944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:56:44 dist-test-slave-j968 krb5kdc[3570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607404, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:56:44.761808 3944 init.cc:377] Logged in from keytab as kudu/127.31.40.3@KRBTEST.COM (short username kudu)
I20260928 14:56:44.763265 3944 fs_manager.cc:359] Metadata directory not provided
I20260928 14:56:44.763367 3944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:56:44.763737 3944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:56:44.767051 3944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/ts-2/data/instance:
uuid: "753ff1c00c62418e9c81bf82cad0a7fe"
format_stamp: "Formatted at 2026-09-28 14:56:44 on dist-test-slave-j968"
server_key: "2dc1e9cb680a158c8f5191bd3826c514"
server_key_iv: "13f3a5f49bbf59ee14f39f7f90bfa5c6"
server_key_version: "encryptionkey@0"
I20260928 14:56:44.768047 3944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/ts-2/wal/instance:
uuid: "753ff1c00c62418e9c81bf82cad0a7fe"
format_stamp: "Formatted at 2026-09-28 14:56:44 on dist-test-slave-j968"
server_key: "2dc1e9cb680a158c8f5191bd3826c514"
server_key_iv: "13f3a5f49bbf59ee14f39f7f90bfa5c6"
server_key_version: "encryptionkey@0"
I20260928 14:56:44.774631 3944 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 14:56:44.778966 3958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:44.780893 3944 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 14:56:44.781103 3944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/ts-2/wal
uuid: "753ff1c00c62418e9c81bf82cad0a7fe"
format_stamp: "Formatted at 2026-09-28 14:56:44 on dist-test-slave-j968"
server_key: "2dc1e9cb680a158c8f5191bd3826c514"
server_key_iv: "13f3a5f49bbf59ee14f39f7f90bfa5c6"
server_key_version: "encryptionkey@0"
I20260928 14:56:44.781339 3944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:56:44.788604 3803 heartbeater.cc:503] Master 127.31.40.62:34795 was elected leader, sending a full tablet report...
I20260928 14:56:44.795612 3944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:56:44.800411 3944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:56:44.800776 3944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:56:44.801846 3944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:56:44.803345 3944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:56:44.803470 3944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:44.803611 3944 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:56:44.803704 3944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:44.839545 3944 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.3:33281
I20260928 14:56:44.839601 4070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.3:33281 every 8 connection(s)
I20260928 14:56:44.840583 3944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
I20260928 14:56:44.846771 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 3944
I20260928 14:56:44.846992 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/ts-2/wal/instance
I20260928 14:56:44.847559 31904 external_mini_cluster.cc:1428] Setting key 07ebc3e142203fa6a57bbb97120cef3e
Sep 28 14:56:44 dist-test-slave-j968 krb5kdc[3570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607404, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.3@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:56:44.863920 4073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:44.843438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:51871 (local address 127.31.40.62:34795)
0928 14:56:44.843993 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:44.843998 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:44.844911 (+ 913us) server_negotiation.cc:407] Connection header received
0928 14:56:44.847222 (+ 2311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:44.847226 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:44.847302 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:44.847395 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:44.849357 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:44.850338 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:44.852425 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:44.852882 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:44.853074 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:44.856998 (+ 3924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:44.857005 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:44.857010 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:44.857055 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:44.859956 (+ 2901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:44.860814 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:44.860818 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:44.860820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:44.860943 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:44.861584 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:44.861588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:44.861590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:44.861931 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:44.862090 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:44.863186 (+ 1096us) server_negotiation.cc:299] Negotiation successful
0928 14:56:44.863435 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":256,"threads_started":1}
I20260928 14:56:44.865046 4074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:44.843841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34795 (local address 127.31.40.3:51871)
0928 14:56:44.844727 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:44.844773 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:44.846812 (+ 2039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:44.847706 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:44.847720 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:44.848469 (+ 749us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:56:44.849148 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:44.849175 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:44.850527 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:44.850535 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:44.852244 (+ 1709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:44.852259 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:44.853189 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:44.853194 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:44.853424 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:44.854145 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:56:44.854170 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:56:44.856690 (+ 2520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:56:44.860144 (+ 3454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:44.860153 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:44.860169 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:44.860688 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:44.861151 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:44.861155 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:44.861158 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:44.861394 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:44.862139 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:56:44.862148 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:56:44.862678 (+ 530us) client_negotiation.cc:769] Sending connection context
0928 14:56:44.863046 (+ 368us) client_negotiation.cc:240] Negotiation successful
0928 14:56:44.863625 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":335,"threads_started":1}
I20260928 14:56:44.866743 4071 heartbeater.cc:347] Connected to a master server at 127.31.40.62:34795
I20260928 14:56:44.867137 4071 heartbeater.cc:464] Registering TS with master...
I20260928 14:56:44.867832 4071 heartbeater.cc:511] Master 127.31.40.62:34795 requested a full tablet report, sending...
I20260928 14:56:44.869544 3614 ts_manager.cc:194] Registered new tserver with Master: 753ff1c00c62418e9c81bf82cad0a7fe (127.31.40.3:33281)
I20260928 14:56:44.870514 3614 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.3@KRBTEST.COM'} at 127.31.40.3:51871
I20260928 14:56:44.877609 31904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:56:44.897079 4073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:44.882922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60456 (local address 127.31.40.62:34795)
0928 14:56:44.883100 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:44.883105 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:44.883588 (+ 483us) server_negotiation.cc:407] Connection header received
0928 14:56:44.883748 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:44.883752 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:44.883821 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:44.883961 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:44.885581 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:44.886623 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:44.888149 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:44.888612 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:44.888722 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:44.891214 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:44.891219 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:44.891223 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:44.891264 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:44.894437 (+ 3173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:44.895153 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:44.895157 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:44.895160 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:44.895283 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:44.895601 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:44.895604 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:44.895607 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:44.896033 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:44.896230 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:44.896641 (+ 411us) server_negotiation.cc:299] Negotiation successful
0928 14:56:44.896830 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 14:56:44.901681 3614 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60456:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:56:44.909426 3614 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:56:44.937718 4084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:44.923113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:42173 (local address 127.0.0.1:42882)
0928 14:56:44.924185 (+ 1072us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:44.924232 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:44.924394 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:44.925960 (+ 1566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:44.925982 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:44.926027 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:56:44.926570 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:44.926588 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:44.929649 (+ 3061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:44.929658 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:44.933148 (+ 3490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:44.933178 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:44.936217 (+ 3039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:44.936229 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:44.936585 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:44.936682 (+ 97us) client_negotiation.cc:769] Sending connection context
0928 14:56:44.936842 (+ 160us) client_negotiation.cc:240] Negotiation successful
0928 14:56:44.937159 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":798,"thread_start_us":627,"threads_started":1}
I20260928 14:56:44.939790 4085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:44.923399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42882 (local address 127.31.40.1:42173)
0928 14:56:44.925093 (+ 1694us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:44.925106 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:44.925146 (+ 40us) server_negotiation.cc:407] Connection header received
0928 14:56:44.925279 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:44.925290 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:44.925632 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:44.926864 (+ 1232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:56:44.927204 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:44.929425 (+ 2221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:44.933541 (+ 4116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:44.935797 (+ 2256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:44.938536 (+ 2739us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:44.938953 (+ 417us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:44.939065 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 14:56:44.939325 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1489,"thread_start_us":1320,"threads_started":1}
I20260928 14:56:44.944716 3738 tablet_service.cc:1467] Processing CreateTablet for tablet d63d49232aa8439c90a07e9379b35238 (DEFAULT_TABLE table=test-table [id=c8ac79f5fa684f33ba23f42c10ff8038]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:56:44.946481 3738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d63d49232aa8439c90a07e9379b35238. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:56:44.948313 4089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:44.925424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37816 (local address 127.31.40.2:37559)
0928 14:56:44.940919 (+ 15495us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:44.940928 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:44.940970 (+ 42us) server_negotiation.cc:407] Connection header received
0928 14:56:44.941080 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:44.941089 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:44.941419 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:44.941647 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:56:44.942132 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:44.943759 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:44.945806 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:44.947178 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:44.947341 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:44.947553 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:44.947774 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 14:56:44.947982 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15345,"thread_start_us":15209,"threads_started":1}
I20260928 14:56:44.948388 4084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:44.924307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.2:37559 (local address 127.0.0.1:37816)
0928 14:56:44.938162 (+ 13855us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:44.938181 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:44.938301 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:44.941650 (+ 3349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:44.941654 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:44.941681 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:56:44.941921 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:44.941937 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:44.943978 (+ 2041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:44.943982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:44.945592 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:44.945626 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:44.947359 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:44.947365 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:44.947543 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:44.947585 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 14:56:44.947717 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 14:56:44.947924 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13747,"mutex_wait_us":1,"thread_start_us":16042,"threads_started":1}
I20260928 14:56:44.949185 4086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:44.923466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.3:33281 (local address 127.0.0.1:37108)
0928 14:56:44.924837 (+ 1371us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:44.924855 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:44.924958 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:44.941001 (+ 16043us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:44.941007 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:44.941041 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:56:44.941418 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:44.941429 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:44.943965 (+ 2536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:44.943971 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:44.946247 (+ 2276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:44.946264 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:44.948280 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:44.948287 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:44.948488 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:44.948521 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 14:56:44.948591 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 14:56:44.948721 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1244,"mutex_wait_us":71,"thread_start_us":2196,"threads_started":1}
I20260928 14:56:44.949805 4088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:44.924851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37108 (local address 127.31.40.3:33281)
0928 14:56:44.940239 (+ 15388us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:44.940250 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:44.940301 (+ 51us) server_negotiation.cc:407] Connection header received
0928 14:56:44.940415 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:44.940424 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:44.940738 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:44.941657 (+ 919us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:56:44.942002 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:44.943757 (+ 1755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:44.946459 (+ 2702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:44.948018 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:44.948934 (+ 916us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:44.949152 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:44.949227 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 14:56:44.949504 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15161,"thread_start_us":16039,"threads_started":1}
I20260928 14:56:44.953693 3872 tablet_service.cc:1467] Processing CreateTablet for tablet d63d49232aa8439c90a07e9379b35238 (DEFAULT_TABLE table=test-table [id=c8ac79f5fa684f33ba23f42c10ff8038]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:56:44.953717 4006 tablet_service.cc:1467] Processing CreateTablet for tablet d63d49232aa8439c90a07e9379b35238 (DEFAULT_TABLE table=test-table [id=c8ac79f5fa684f33ba23f42c10ff8038]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:56:44.954994 3872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d63d49232aa8439c90a07e9379b35238. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:56:44.955109 4006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d63d49232aa8439c90a07e9379b35238. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:56:44.962568 4090 tablet_bootstrap.cc:492] T d63d49232aa8439c90a07e9379b35238 P c224246c15ff4c8587a6342fb5d3973f: Bootstrap starting.
I20260928 14:56:44.966622 4090 tablet_bootstrap.cc:654] T d63d49232aa8439c90a07e9379b35238 P c224246c15ff4c8587a6342fb5d3973f: Neither blocks nor log segments found. Creating new log.
I20260928 14:56:44.968257 4090 log.cc:824] T d63d49232aa8439c90a07e9379b35238 P c224246c15ff4c8587a6342fb5d3973f: Log is configured to *not* fsync() on all Append() calls
I20260928 14:56:44.968706 4091 tablet_bootstrap.cc:492] T d63d49232aa8439c90a07e9379b35238 P 51ac507dc8c4475aaf5982433257c582: Bootstrap starting.
I20260928 14:56:44.972615 4090 tablet_bootstrap.cc:492] T d63d49232aa8439c90a07e9379b35238 P c224246c15ff4c8587a6342fb5d3973f: No bootstrap required, opened a new log
I20260928 14:56:44.972858 4091 tablet_bootstrap.cc:654] T d63d49232aa8439c90a07e9379b35238 P 51ac507dc8c4475aaf5982433257c582: Neither blocks nor log segments found. Creating new log.
I20260928 14:56:44.972921 4090 ts_tablet_manager.cc:1397] T d63d49232aa8439c90a07e9379b35238 P c224246c15ff4c8587a6342fb5d3973f: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260928 14:56:44.974011 4091 log.cc:824] T d63d49232aa8439c90a07e9379b35238 P 51ac507dc8c4475aaf5982433257c582: Log is configured to *not* fsync() on all Append() calls
I20260928 14:56:44.976877 4092 tablet_bootstrap.cc:492] T d63d49232aa8439c90a07e9379b35238 P 753ff1c00c62418e9c81bf82cad0a7fe: Bootstrap starting.
I20260928 14:56:44.977134 4091 tablet_bootstrap.cc:492] T d63d49232aa8439c90a07e9379b35238 P 51ac507dc8c4475aaf5982433257c582: No bootstrap required, opened a new log
I20260928 14:56:44.977873 4091 ts_tablet_manager.cc:1397] T d63d49232aa8439c90a07e9379b35238 P 51ac507dc8c4475aaf5982433257c582: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260928 14:56:44.980844 4092 tablet_bootstrap.cc:654] T d63d49232aa8439c90a07e9379b35238 P 753ff1c00c62418e9c81bf82cad0a7fe: Neither blocks nor log segments found. Creating new log.
I20260928 14:56:44.982173 4092 log.cc:824] T d63d49232aa8439c90a07e9379b35238 P 753ff1c00c62418e9c81bf82cad0a7fe: Log is configured to *not* fsync() on all Append() calls
I20260928 14:56:44.982450 4090 raft_consensus.cc:348] T d63d49232aa8439c90a07e9379b35238 P c224246c15ff4c8587a6342fb5d3973f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c224246c15ff4c8587a6342fb5d3973f" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 42173 } } peers { permanent_uuid: "753ff1c00c62418e9c81bf82cad0a7fe" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33281 } } peers { permanent_uuid: "51ac507dc8c4475aaf5982433257c582" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 37559 } }
I20260928 14:56:44.983088 4090 raft_consensus.cc:374] T d63d49232aa8439c90a07e9379b35238 P c224246c15ff4c8587a6342fb5d3973f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:56:44.983209 4090 raft_consensus.cc:729] T d63d49232aa8439c90a07e9379b35238 P c224246c15ff4c8587a6342fb5d3973f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c224246c15ff4c8587a6342fb5d3973f, State: Initialized, Role: FOLLOWER
I20260928 14:56:44.983958 4090 consensus_queue.cc:260] T d63d49232aa8439c90a07e9379b35238 P c224246c15ff4c8587a6342fb5d3973f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c224246c15ff4c8587a6342fb5d3973f" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 42173 } } peers { permanent_uuid: "753ff1c00c62418e9c81bf82cad0a7fe" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33281 } } peers { permanent_uuid: "51ac507dc8c4475aaf5982433257c582" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 37559 } }
I20260928 14:56:44.986027 4092 tablet_bootstrap.cc:492] T d63d49232aa8439c90a07e9379b35238 P 753ff1c00c62418e9c81bf82cad0a7fe: No bootstrap required, opened a new log
I20260928 14:56:44.986420 4090 ts_tablet_manager.cc:1428] T d63d49232aa8439c90a07e9379b35238 P c224246c15ff4c8587a6342fb5d3973f: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260928 14:56:44.986734 4092 ts_tablet_manager.cc:1397] T d63d49232aa8439c90a07e9379b35238 P 753ff1c00c62418e9c81bf82cad0a7fe: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.000s
I20260928 14:56:44.987195 4091 raft_consensus.cc:348] T d63d49232aa8439c90a07e9379b35238 P 51ac507dc8c4475aaf5982433257c582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c224246c15ff4c8587a6342fb5d3973f" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 42173 } } peers { permanent_uuid: "753ff1c00c62418e9c81bf82cad0a7fe" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33281 } } peers { permanent_uuid: "51ac507dc8c4475aaf5982433257c582" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 37559 } }
I20260928 14:56:44.987671 4091 raft_consensus.cc:374] T d63d49232aa8439c90a07e9379b35238 P 51ac507dc8c4475aaf5982433257c582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:56:44.987775 4091 raft_consensus.cc:729] T d63d49232aa8439c90a07e9379b35238 P 51ac507dc8c4475aaf5982433257c582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51ac507dc8c4475aaf5982433257c582, State: Initialized, Role: FOLLOWER
I20260928 14:56:44.988467 4091 consensus_queue.cc:260] T d63d49232aa8439c90a07e9379b35238 P 51ac507dc8c4475aaf5982433257c582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c224246c15ff4c8587a6342fb5d3973f" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 42173 } } peers { permanent_uuid: "753ff1c00c62418e9c81bf82cad0a7fe" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33281 } } peers { permanent_uuid: "51ac507dc8c4475aaf5982433257c582" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 37559 } }
I20260928 14:56:44.989595 3938 heartbeater.cc:503] Master 127.31.40.62:34795 was elected leader, sending a full tablet report...
I20260928 14:56:44.990288 4091 ts_tablet_manager.cc:1428] T d63d49232aa8439c90a07e9379b35238 P 51ac507dc8c4475aaf5982433257c582: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260928 14:56:44.995949 4092 raft_consensus.cc:348] T d63d49232aa8439c90a07e9379b35238 P 753ff1c00c62418e9c81bf82cad0a7fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c224246c15ff4c8587a6342fb5d3973f" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 42173 } } peers { permanent_uuid: "753ff1c00c62418e9c81bf82cad0a7fe" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33281 } } peers { permanent_uuid: "51ac507dc8c4475aaf5982433257c582" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 37559 } }
I20260928 14:56:44.996383 4092 raft_consensus.cc:374] T d63d49232aa8439c90a07e9379b35238 P 753ff1c00c62418e9c81bf82cad0a7fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:56:44.996490 4092 raft_consensus.cc:729] T d63d49232aa8439c90a07e9379b35238 P 753ff1c00c62418e9c81bf82cad0a7fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 753ff1c00c62418e9c81bf82cad0a7fe, State: Initialized, Role: FOLLOWER
I20260928 14:56:44.997113 4092 consensus_queue.cc:260] T d63d49232aa8439c90a07e9379b35238 P 753ff1c00c62418e9c81bf82cad0a7fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c224246c15ff4c8587a6342fb5d3973f" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 42173 } } peers { permanent_uuid: "753ff1c00c62418e9c81bf82cad0a7fe" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33281 } } peers { permanent_uuid: "51ac507dc8c4475aaf5982433257c582" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 37559 } }
I20260928 14:56:44.998190 4071 heartbeater.cc:503] Master 127.31.40.62:34795 was elected leader, sending a full tablet report...
I20260928 14:56:44.998647 4092 ts_tablet_manager.cc:1428] T d63d49232aa8439c90a07e9379b35238 P 753ff1c00c62418e9c81bf82cad0a7fe: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
W20260928 14:56:45.005635 3804 tablet.cc:2369] T d63d49232aa8439c90a07e9379b35238 P c224246c15ff4c8587a6342fb5d3973f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:56:45.017630 4099 raft_consensus.cc:482] T d63d49232aa8439c90a07e9379b35238 P 753ff1c00c62418e9c81bf82cad0a7fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:56:45.017983 4099 raft_consensus.cc:504] T d63d49232aa8439c90a07e9379b35238 P 753ff1c00c62418e9c81bf82cad0a7fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c224246c15ff4c8587a6342fb5d3973f" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 42173 } } peers { permanent_uuid: "753ff1c00c62418e9c81bf82cad0a7fe" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33281 } } peers { permanent_uuid: "51ac507dc8c4475aaf5982433257c582" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 37559 } }
I20260928 14:56:45.020002 4099 leader_election.cc:288] T d63d49232aa8439c90a07e9379b35238 P 753ff1c00c62418e9c81bf82cad0a7fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c224246c15ff4c8587a6342fb5d3973f (127.31.40.1:42173), 51ac507dc8c4475aaf5982433257c582 (127.31.40.2:37559)
I20260928 14:56:45.027449 4089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:45.020685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:58071 (local address 127.31.40.2:37559)
0928 14:56:45.020906 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:45.020915 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:45.020939 (+ 24us) server_negotiation.cc:407] Connection header received
0928 14:56:45.021035 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:45.021043 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:45.021149 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:45.021268 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:56:45.021753 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:45.022906 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:45.025055 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:45.026375 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:45.026486 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:45.026597 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:45.027013 (+ 416us) server_negotiation.cc:299] Negotiation successful
0928 14:56:45.027164 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 14:56:45.027474 4074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:45.020343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.2:37559 (local address 127.31.40.3:58071)
0928 14:56:45.020687 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:45.020710 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:45.020872 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:45.021308 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:45.021311 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:45.021341 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:56:45.021586 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:45.021598 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:45.023068 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:45.023074 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:45.024874 (+ 1800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:45.024890 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:45.026552 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:45.026559 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:45.026812 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:45.026853 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 14:56:45.026975 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 14:56:45.027131 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":4}
I20260928 14:56:45.028771 4085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:45.020406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:41683 (local address 127.31.40.1:42173)
0928 14:56:45.020764 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:45.020772 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:45.021500 (+ 728us) server_negotiation.cc:407] Connection header received
0928 14:56:45.021658 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:45.021666 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:45.021753 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:45.021821 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:56:45.022569 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:45.023742 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:45.026146 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:45.027456 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:45.027612 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:45.027734 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:45.028343 (+ 609us) server_negotiation.cc:299] Negotiation successful
0928 14:56:45.028485 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181}
I20260928 14:56:45.028895 4100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:45.020447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:42173 (local address 127.31.40.3:41683)
0928 14:56:45.021370 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:45.021393 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:45.021520 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:45.021891 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:45.021896 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:45.021925 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:56:45.022344 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:45.022358 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:45.023987 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:45.023991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:45.025947 (+ 1956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:45.025965 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:45.027948 (+ 1983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:45.027965 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:45.028161 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:45.028208 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 14:56:45.028309 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 14:56:45.028468 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":794,"mutex_wait_us":1,"thread_start_us":579,"threads_started":1}
I20260928 14:56:45.028785 3893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d63d49232aa8439c90a07e9379b35238" candidate_uuid: "753ff1c00c62418e9c81bf82cad0a7fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51ac507dc8c4475aaf5982433257c582" is_pre_election: true
I20260928 14:56:45.029433 3893 raft_consensus.cc:2454] T d63d49232aa8439c90a07e9379b35238 P 51ac507dc8c4475aaf5982433257c582 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 753ff1c00c62418e9c81bf82cad0a7fe in term 0.
I20260928 14:56:45.030005 3758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d63d49232aa8439c90a07e9379b35238" candidate_uuid: "753ff1c00c62418e9c81bf82cad0a7fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c224246c15ff4c8587a6342fb5d3973f" is_pre_election: true
I20260928 14:56:45.030224 3962 leader_election.cc:302] T d63d49232aa8439c90a07e9379b35238 P 753ff1c00c62418e9c81bf82cad0a7fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51ac507dc8c4475aaf5982433257c582, 753ff1c00c62418e9c81bf82cad0a7fe; no voters:
I20260928 14:56:45.030618 3758 raft_consensus.cc:2454] T d63d49232aa8439c90a07e9379b35238 P c224246c15ff4c8587a6342fb5d3973f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 753ff1c00c62418e9c81bf82cad0a7fe in term 0.
I20260928 14:56:45.030685 4099 raft_consensus.cc:2781] T d63d49232aa8439c90a07e9379b35238 P 753ff1c00c62418e9c81bf82cad0a7fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:56:45.030853 4099 raft_consensus.cc:482] T d63d49232aa8439c90a07e9379b35238 P 753ff1c00c62418e9c81bf82cad0a7fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:56:45.030964 4099 raft_consensus.cc:3037] T d63d49232aa8439c90a07e9379b35238 P 753ff1c00c62418e9c81bf82cad0a7fe [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:56:45.033607 4099 raft_consensus.cc:504] T d63d49232aa8439c90a07e9379b35238 P 753ff1c00c62418e9c81bf82cad0a7fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c224246c15ff4c8587a6342fb5d3973f" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 42173 } } peers { permanent_uuid: "753ff1c00c62418e9c81bf82cad0a7fe" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33281 } } peers { permanent_uuid: "51ac507dc8c4475aaf5982433257c582" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 37559 } }
I20260928 14:56:45.034451 4099 leader_election.cc:288] T d63d49232aa8439c90a07e9379b35238 P 753ff1c00c62418e9c81bf82cad0a7fe [CANDIDATE]: Term 1 election: Requested vote from peers c224246c15ff4c8587a6342fb5d3973f (127.31.40.1:42173), 51ac507dc8c4475aaf5982433257c582 (127.31.40.2:37559)
I20260928 14:56:45.034929 3758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d63d49232aa8439c90a07e9379b35238" candidate_uuid: "753ff1c00c62418e9c81bf82cad0a7fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c224246c15ff4c8587a6342fb5d3973f"
I20260928 14:56:45.035158 3893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d63d49232aa8439c90a07e9379b35238" candidate_uuid: "753ff1c00c62418e9c81bf82cad0a7fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51ac507dc8c4475aaf5982433257c582"
I20260928 14:56:45.035218 3758 raft_consensus.cc:3037] T d63d49232aa8439c90a07e9379b35238 P c224246c15ff4c8587a6342fb5d3973f [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:56:45.035399 3893 raft_consensus.cc:3037] T d63d49232aa8439c90a07e9379b35238 P 51ac507dc8c4475aaf5982433257c582 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:56:45.037796 3893 raft_consensus.cc:2454] T d63d49232aa8439c90a07e9379b35238 P 51ac507dc8c4475aaf5982433257c582 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 753ff1c00c62418e9c81bf82cad0a7fe in term 1.
I20260928 14:56:45.037804 3758 raft_consensus.cc:2454] T d63d49232aa8439c90a07e9379b35238 P c224246c15ff4c8587a6342fb5d3973f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 753ff1c00c62418e9c81bf82cad0a7fe in term 1.
I20260928 14:56:45.038308 3962 leader_election.cc:302] T d63d49232aa8439c90a07e9379b35238 P 753ff1c00c62418e9c81bf82cad0a7fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51ac507dc8c4475aaf5982433257c582, 753ff1c00c62418e9c81bf82cad0a7fe; no voters:
I20260928 14:56:45.038673 4099 raft_consensus.cc:2781] T d63d49232aa8439c90a07e9379b35238 P 753ff1c00c62418e9c81bf82cad0a7fe [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:56:45.039209 4099 raft_consensus.cc:686] T d63d49232aa8439c90a07e9379b35238 P 753ff1c00c62418e9c81bf82cad0a7fe [term 1 LEADER]: Becoming Leader. State: Replica: 753ff1c00c62418e9c81bf82cad0a7fe, State: Running, Role: LEADER
I20260928 14:56:45.039812 4099 consensus_queue.cc:237] T d63d49232aa8439c90a07e9379b35238 P 753ff1c00c62418e9c81bf82cad0a7fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c224246c15ff4c8587a6342fb5d3973f" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 42173 } } peers { permanent_uuid: "753ff1c00c62418e9c81bf82cad0a7fe" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33281 } } peers { permanent_uuid: "51ac507dc8c4475aaf5982433257c582" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 37559 } }
I20260928 14:56:45.045457 3613 catalog_manager.cc:5495] T d63d49232aa8439c90a07e9379b35238 P 753ff1c00c62418e9c81bf82cad0a7fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 753ff1c00c62418e9c81bf82cad0a7fe (127.31.40.3). New cstate: current_term: 1 leader_uuid: "753ff1c00c62418e9c81bf82cad0a7fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c224246c15ff4c8587a6342fb5d3973f" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 42173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "753ff1c00c62418e9c81bf82cad0a7fe" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33281 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51ac507dc8c4475aaf5982433257c582" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 37559 } health_report { overall_health: UNKNOWN } } }
W20260928 14:56:45.051326 3939 tablet.cc:2369] T d63d49232aa8439c90a07e9379b35238 P 51ac507dc8c4475aaf5982433257c582: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:56:45.067222 4088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:45.060746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37110 (local address 127.31.40.3:33281)
0928 14:56:45.060967 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:45.060973 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:45.061247 (+ 274us) server_negotiation.cc:407] Connection header received
0928 14:56:45.061492 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:45.061498 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:45.061588 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:45.061787 (+ 199us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:56:45.062224 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:45.063102 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:45.064644 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:45.065219 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:45.065325 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:45.066006 (+ 681us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:56:45.066603 (+ 597us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:56:45.066724 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:45.066833 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 14:56:45.066905 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 14:56:45.076411 3758 raft_consensus.cc:1261] T d63d49232aa8439c90a07e9379b35238 P c224246c15ff4c8587a6342fb5d3973f [term 1 FOLLOWER]: Refusing update from remote peer 753ff1c00c62418e9c81bf82cad0a7fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:56:45.076411 3893 raft_consensus.cc:1261] T d63d49232aa8439c90a07e9379b35238 P 51ac507dc8c4475aaf5982433257c582 [term 1 FOLLOWER]: Refusing update from remote peer 753ff1c00c62418e9c81bf82cad0a7fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:56:45.077838 4101 consensus_queue.cc:1035] T d63d49232aa8439c90a07e9379b35238 P 753ff1c00c62418e9c81bf82cad0a7fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "c224246c15ff4c8587a6342fb5d3973f" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 42173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:56:45.078318 4099 consensus_queue.cc:1035] T d63d49232aa8439c90a07e9379b35238 P 753ff1c00c62418e9c81bf82cad0a7fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "51ac507dc8c4475aaf5982433257c582" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 37559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:56:45.089607 4104 mvcc.cc:205] Tried to move back new op lower bound from 7334327931179991040 to 7334327931047927808. Current Snapshot: MvccSnapshot[applied={T|T < 7334327931179991040}]
I20260928 14:56:45.090430 4105 mvcc.cc:205] Tried to move back new op lower bound from 7334327931179991040 to 7334327931047927808. Current Snapshot: MvccSnapshot[applied={T|T < 7334327931179991040}]
I20260928 14:56:45.094326 4107 mvcc.cc:205] Tried to move back new op lower bound from 7334327931179991040 to 7334327931047927808. Current Snapshot: MvccSnapshot[applied={T|T < 7334327931179991040}]
I20260928 14:56:45.105962 3614 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:56:45.106297 3614 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:56:45.110488 3614 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 787b318e05c44c40a5f16295168835e6: Sending DeleteTablet for 3 replicas of tablet d63d49232aa8439c90a07e9379b35238
I20260928 14:56:45.111680 4006 tablet_service.cc:1514] Processing DeleteTablet for tablet d63d49232aa8439c90a07e9379b35238 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:56:45 UTC) from {username='kudu', principal='kudu/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:37108
I20260928 14:56:45.111718 3738 tablet_service.cc:1514] Processing DeleteTablet for tablet d63d49232aa8439c90a07e9379b35238 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:56:45 UTC) from {username='kudu', principal='kudu/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:42882
I20260928 14:56:45.112854 4113 tablet_replica.cc:330] T d63d49232aa8439c90a07e9379b35238 P c224246c15ff4c8587a6342fb5d3973f: stopping tablet replica
I20260928 14:56:45.113544 4113 raft_consensus.cc:2229] T d63d49232aa8439c90a07e9379b35238 P c224246c15ff4c8587a6342fb5d3973f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:56:45.114276 4113 raft_consensus.cc:2258] T d63d49232aa8439c90a07e9379b35238 P c224246c15ff4c8587a6342fb5d3973f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:56:45.115557 3872 tablet_service.cc:1514] Processing DeleteTablet for tablet d63d49232aa8439c90a07e9379b35238 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:56:45 UTC) from {username='kudu', principal='kudu/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:37816
I20260928 14:56:45.117061 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 3676
I20260928 14:56:45.123394 4114 tablet_replica.cc:330] T d63d49232aa8439c90a07e9379b35238 P 753ff1c00c62418e9c81bf82cad0a7fe: stopping tablet replica
I20260928 14:56:45.124181 4114 raft_consensus.cc:2229] T d63d49232aa8439c90a07e9379b35238 P 753ff1c00c62418e9c81bf82cad0a7fe [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:56:45.125181 4114 raft_consensus.cc:2258] T d63d49232aa8439c90a07e9379b35238 P 753ff1c00c62418e9c81bf82cad0a7fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:56:45.127987 4114 ts_tablet_manager.cc:1905] T d63d49232aa8439c90a07e9379b35238 P 753ff1c00c62418e9c81bf82cad0a7fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:56:45.129720 4113 ts_tablet_manager.cc:1905] T d63d49232aa8439c90a07e9379b35238 P c224246c15ff4c8587a6342fb5d3973f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:56:45.132445 4115 tablet_replica.cc:330] T d63d49232aa8439c90a07e9379b35238 P 51ac507dc8c4475aaf5982433257c582: stopping tablet replica
I20260928 14:56:45.132979 4115 raft_consensus.cc:2229] T d63d49232aa8439c90a07e9379b35238 P 51ac507dc8c4475aaf5982433257c582 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:56:45.133942 4115 raft_consensus.cc:2258] T d63d49232aa8439c90a07e9379b35238 P 51ac507dc8c4475aaf5982433257c582 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:56:45.136715 4115 ts_tablet_manager.cc:1905] T d63d49232aa8439c90a07e9379b35238 P 51ac507dc8c4475aaf5982433257c582: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 14:56:45 dist-test-slave-j968 krb5kdc[3570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.1@KRBTEST.COM
I20260928 14:56:45.137738 4113 ts_tablet_manager.cc:1918] T d63d49232aa8439c90a07e9379b35238 P c224246c15ff4c8587a6342fb5d3973f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:56:45.138015 4113 log.cc:1196] T d63d49232aa8439c90a07e9379b35238 P c224246c15ff4c8587a6342fb5d3973f: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/ts-0/wal/wals/d63d49232aa8439c90a07e9379b35238
I20260928 14:56:45.138664 4113 ts_tablet_manager.cc:1939] T d63d49232aa8439c90a07e9379b35238 P c224246c15ff4c8587a6342fb5d3973f: Deleting consensus metadata
I20260928 14:56:45.140964 3598 catalog_manager.cc:4841] TS c224246c15ff4c8587a6342fb5d3973f (127.31.40.1:42173): tablet d63d49232aa8439c90a07e9379b35238 (table test-table [id=c8ac79f5fa684f33ba23f42c10ff8038]) successfully deleted
I20260928 14:56:45.143430 4114 ts_tablet_manager.cc:1918] T d63d49232aa8439c90a07e9379b35238 P 753ff1c00c62418e9c81bf82cad0a7fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:56:45.143735 4114 log.cc:1196] T d63d49232aa8439c90a07e9379b35238 P 753ff1c00c62418e9c81bf82cad0a7fe: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/ts-2/wal/wals/d63d49232aa8439c90a07e9379b35238
I20260928 14:56:45.144418 4114 ts_tablet_manager.cc:1939] T d63d49232aa8439c90a07e9379b35238 P 753ff1c00c62418e9c81bf82cad0a7fe: Deleting consensus metadata
I20260928 14:56:45.146505 3599 catalog_manager.cc:4841] TS 753ff1c00c62418e9c81bf82cad0a7fe (127.31.40.3:33281): tablet d63d49232aa8439c90a07e9379b35238 (table test-table [id=c8ac79f5fa684f33ba23f42c10ff8038]) successfully deleted
I20260928 14:56:45.146854 4085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:45.119170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42894 (local address 127.31.40.1:42173)
0928 14:56:45.119411 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:45.119420 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:45.126189 (+ 6769us) server_negotiation.cc:407] Connection header received
0928 14:56:45.126406 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:45.126412 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:45.126497 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:45.126605 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:45.128439 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:45.129646 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:45.131637 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:45.132424 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:45.132550 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:45.138372 (+ 5822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:45.138389 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:45.138409 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:45.138463 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:45.142431 (+ 3968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:45.143428 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:45.143438 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:45.143441 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:45.143627 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:45.144178 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:45.144184 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:45.144187 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:45.144681 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:45.144983 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:45.145673 (+ 690us) server_negotiation.cc:299] Negotiation successful
0928 14:56:45.145936 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 14:56:45.147528 4115 ts_tablet_manager.cc:1918] T d63d49232aa8439c90a07e9379b35238 P 51ac507dc8c4475aaf5982433257c582: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:56:45.147734 4115 log.cc:1196] T d63d49232aa8439c90a07e9379b35238 P 51ac507dc8c4475aaf5982433257c582: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607365362981-31904-0/minicluster-data/ts-1/wal/wals/d63d49232aa8439c90a07e9379b35238
I20260928 14:56:45.148115 3798 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:56:45.148386 4115 ts_tablet_manager.cc:1939] T d63d49232aa8439c90a07e9379b35238 P 51ac507dc8c4475aaf5982433257c582: Deleting consensus metadata
I20260928 14:56:45.151830 3601 catalog_manager.cc:4841] TS 51ac507dc8c4475aaf5982433257c582 (127.31.40.2:37559): tablet d63d49232aa8439c90a07e9379b35238 (table test-table [id=c8ac79f5fa684f33ba23f42c10ff8038]) successfully deleted
I20260928 14:56:45.271032 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 3676
I20260928 14:56:45.290292 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 3809
Sep 28 14:56:45 dist-test-slave-j968 krb5kdc[3570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.2@KRBTEST.COM
I20260928 14:56:45.307181 4089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:45.291186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37830 (local address 127.31.40.2:37559)
0928 14:56:45.291366 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:45.291374 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:45.291396 (+ 22us) server_negotiation.cc:407] Connection header received
0928 14:56:45.291702 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:45.291709 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:45.291819 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:45.292005 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:45.293265 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:45.294057 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:45.295316 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:45.295754 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:45.295918 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:45.300171 (+ 4253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:45.300184 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:45.300200 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:45.300242 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:45.303647 (+ 3405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:45.304475 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:45.304490 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:45.304492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:45.304635 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:45.305197 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:45.305206 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:45.305210 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:45.305729 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:45.306215 (+ 486us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:45.306451 (+ 236us) server_negotiation.cc:299] Negotiation successful
0928 14:56:45.306717 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 14:56:45.308112 3933 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:56:45.408649 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 3809
I20260928 14:56:45.427253 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 3944
Sep 28 14:56:45 dist-test-slave-j968 krb5kdc[3570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.3@KRBTEST.COM
I20260928 14:56:45.445731 4088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:45.428127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37122 (local address 127.31.40.3:33281)
0928 14:56:45.428340 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:45.428346 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:45.428369 (+ 23us) server_negotiation.cc:407] Connection header received
0928 14:56:45.428524 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:45.428530 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:45.428615 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:45.428733 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:45.430197 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:45.431452 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:45.432884 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:45.433594 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:45.433744 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:45.438427 (+ 4683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:45.438445 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:45.438467 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:45.438524 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:45.442432 (+ 3908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:45.443250 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:45.443258 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:45.443260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:45.443430 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:45.444022 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:45.444026 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:45.444028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:45.444475 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:45.444805 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:45.445113 (+ 308us) server_negotiation.cc:299] Negotiation successful
0928 14:56:45.445339 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 14:56:45.446564 4066 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:56:45.570717 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 3944
I20260928 14:56:45.592370 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 3583
I20260928 14:56:45.593653 3645 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:56:45.703737 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 3583
2026-09-28T14:56:45Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4470 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T14:56:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:56:45Z Disabled control of system clock
I20260928 14:56:45.775964 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:35669
--webserver_interface=127.31.40.62
--webserver_port=0
--builtin_ntp_servers=127.31.40.20:32913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.40.62:35669
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 14:56:46.119454 4124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:56:46.119777 4124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:56:46.119937 4124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:56:46.127497 4124 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:56:46.127619 4124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:56:46.127658 4124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:56:46.127709 4124 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:56:46.127754 4124 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:56:46.141029 4124 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:32913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.40.62:35669
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:35669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:56:46.142740 4124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:56:46.145334 4124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:56:46.158044 4129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:46.158016 4130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:46.158615 4132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:46.158735 4124 server_base.cc:1034] running on GCE node
I20260928 14:56:46.159592 4124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:56:46.161170 4124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:56:46.162387 4124 hybrid_clock.cc:648] HybridClock initialized: now 1790607406162355 us; error 55 us; skew 500 ppm
I20260928 14:56:46.162804 4124 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 14:56:46.165290 4124 webserver.cc:460] Webserver started at http://127.31.40.62:38949/ using document root <none> and password file <none>
I20260928 14:56:46.166018 4124 fs_manager.cc:359] Metadata directory not provided
I20260928 14:56:46.166105 4124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:56:46.166440 4124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:56:46.169462 4124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/master-0/data/instance:
uuid: "a7a9429a791946db92674624cd83b760"
format_stamp: "Formatted at 2026-09-28 14:56:46 on dist-test-slave-j968"
server_key: "dd4943d083c84e8e9fc12dd83d4cf84e"
server_key_iv: "773ad9e6fe27827e0f37fe2b8ac5d9c4"
server_key_version: "encryptionkey@0"
I20260928 14:56:46.170305 4124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/master-0/wal/instance:
uuid: "a7a9429a791946db92674624cd83b760"
format_stamp: "Formatted at 2026-09-28 14:56:46 on dist-test-slave-j968"
server_key: "dd4943d083c84e8e9fc12dd83d4cf84e"
server_key_iv: "773ad9e6fe27827e0f37fe2b8ac5d9c4"
server_key_version: "encryptionkey@0"
I20260928 14:56:46.175458 4124 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 14:56:46.179011 4138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:46.180502 4124 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 14:56:46.180676 4124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/master-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/master-0/wal
uuid: "a7a9429a791946db92674624cd83b760"
format_stamp: "Formatted at 2026-09-28 14:56:46 on dist-test-slave-j968"
server_key: "dd4943d083c84e8e9fc12dd83d4cf84e"
server_key_iv: "773ad9e6fe27827e0f37fe2b8ac5d9c4"
server_key_version: "encryptionkey@0"
I20260928 14:56:46.180881 4124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:56:46.186889 4124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:56:46.187732 4124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:56:46.188077 4124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:56:46.207165 4124 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.62:35669
I20260928 14:56:46.207206 4190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.62:35669 every 8 connection(s)
I20260928 14:56:46.208652 4124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
I20260928 14:56:46.211887 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 4124
I20260928 14:56:46.212105 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/master-0/wal/instance
I20260928 14:56:46.212914 31904 external_mini_cluster.cc:1428] Setting key f76369faa9e264a4b5eb07f21766d264
I20260928 14:56:46.213249 4191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:56:46.223021 4194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:46.215048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50044 (local address 127.31.40.62:35669)
0928 14:56:46.216218 (+ 1170us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:46.216229 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:46.216285 (+ 56us) server_negotiation.cc:407] Connection header received
0928 14:56:46.218812 (+ 2527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:46.218869 (+ 57us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:46.219468 (+ 599us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:46.219960 (+ 492us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:46.221229 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:46.221235 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:46.221253 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:56:46.221273 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:46.221339 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:56:46.221357 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:46.221555 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:46.221661 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 14:56:46.222013 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":434,"threads_started":1}
I20260928 14:56:46.227662 4191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7a9429a791946db92674624cd83b760: Bootstrap starting.
I20260928 14:56:46.230871 4191 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7a9429a791946db92674624cd83b760: Neither blocks nor log segments found. Creating new log.
I20260928 14:56:46.232118 4191 log.cc:824] T 00000000000000000000000000000000 P a7a9429a791946db92674624cd83b760: Log is configured to *not* fsync() on all Append() calls
I20260928 14:56:46.235229 4191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7a9429a791946db92674624cd83b760: No bootstrap required, opened a new log
I20260928 14:56:46.243230 4191 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7a9429a791946db92674624cd83b760 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7a9429a791946db92674624cd83b760" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 35669 } }
I20260928 14:56:46.243702 4191 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7a9429a791946db92674624cd83b760 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:56:46.243826 4191 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7a9429a791946db92674624cd83b760 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7a9429a791946db92674624cd83b760, State: Initialized, Role: FOLLOWER
I20260928 14:56:46.244515 4191 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7a9429a791946db92674624cd83b760 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7a9429a791946db92674624cd83b760" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 35669 } }
I20260928 14:56:46.244773 4191 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7a9429a791946db92674624cd83b760 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:56:46.244922 4191 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7a9429a791946db92674624cd83b760 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:56:46.245163 4191 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7a9429a791946db92674624cd83b760 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:56:46.247404 4191 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7a9429a791946db92674624cd83b760 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7a9429a791946db92674624cd83b760" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 35669 } }
I20260928 14:56:46.248075 4191 leader_election.cc:302] T 00000000000000000000000000000000 P a7a9429a791946db92674624cd83b760 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7a9429a791946db92674624cd83b760; no voters:
I20260928 14:56:46.248747 4191 leader_election.cc:288] T 00000000000000000000000000000000 P a7a9429a791946db92674624cd83b760 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:56:46.248986 4196 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7a9429a791946db92674624cd83b760 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:56:46.249779 4196 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7a9429a791946db92674624cd83b760 [term 1 LEADER]: Becoming Leader. State: Replica: a7a9429a791946db92674624cd83b760, State: Running, Role: LEADER
I20260928 14:56:46.250351 4196 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7a9429a791946db92674624cd83b760 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7a9429a791946db92674624cd83b760" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 35669 } }
I20260928 14:56:46.251112 4191 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7a9429a791946db92674624cd83b760 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:56:46.253412 4197 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7a9429a791946db92674624cd83b760 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7a9429a791946db92674624cd83b760" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7a9429a791946db92674624cd83b760" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 35669 } } }
I20260928 14:56:46.253571 4198 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7a9429a791946db92674624cd83b760 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7a9429a791946db92674624cd83b760. Latest consensus state: current_term: 1 leader_uuid: "a7a9429a791946db92674624cd83b760" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7a9429a791946db92674624cd83b760" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 35669 } } }
I20260928 14:56:46.253841 4197 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7a9429a791946db92674624cd83b760 [sys.catalog]: This master's current role is: LEADER
I20260928 14:56:46.253872 4198 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7a9429a791946db92674624cd83b760 [sys.catalog]: This master's current role is: LEADER
I20260928 14:56:46.256599 4204 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:56:46.262570 4204 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:56:46.272342 4204 catalog_manager.cc:1347] Generated new cluster ID: 16e7690d72f7447b8fe995554c9c94c2
I20260928 14:56:46.272481 4204 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:56:46.291908 4204 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:56:46.292439 4204 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:56:46.306541 4204 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7a9429a791946db92674624cd83b760: Generated new TSK 0
I20260928 14:56:46.307734 4204 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:56:46.327330 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.1:0
--local_ip_for_outbound_sockets=127.31.40.1
--webserver_interface=127.31.40.1
--webserver_port=0
--tserver_master_addrs=127.31.40.62:35669
--builtin_ntp_servers=127.31.40.20:32913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 14:56:46.688493 4215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:56:46.688860 4215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:56:46.689033 4215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:56:46.696803 4215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:56:46.696918 4215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:56:46.697084 4215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.1
I20260928 14:56:46.707069 4215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:32913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.40.1
--webserver_port=0
--tserver_master_addrs=127.31.40.62:35669
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.1
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:56:46.708529 4215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:56:46.710501 4215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:56:46.725108 4223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:46.724889 4221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:46.725826 4220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:46.729750 4215 server_base.cc:1034] running on GCE node
I20260928 14:56:46.730636 4215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:56:46.731937 4215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:56:46.733213 4215 hybrid_clock.cc:648] HybridClock initialized: now 1790607406733127 us; error 81 us; skew 500 ppm
I20260928 14:56:46.733711 4215 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 14:56:46.736415 4215 webserver.cc:460] Webserver started at http://127.31.40.1:42699/ using document root <none> and password file <none>
I20260928 14:56:46.737213 4215 fs_manager.cc:359] Metadata directory not provided
I20260928 14:56:46.737316 4215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:56:46.737815 4215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:56:46.741569 4215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/ts-0/data/instance:
uuid: "fcf59744cadd4bb989354022b1ea1756"
format_stamp: "Formatted at 2026-09-28 14:56:46 on dist-test-slave-j968"
server_key: "c91f972cd02ace9fdb7f64ecbcff992a"
server_key_iv: "595c160b676a72614927ce4b8d56d111"
server_key_version: "encryptionkey@0"
I20260928 14:56:46.742410 4215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/ts-0/wal/instance:
uuid: "fcf59744cadd4bb989354022b1ea1756"
format_stamp: "Formatted at 2026-09-28 14:56:46 on dist-test-slave-j968"
server_key: "c91f972cd02ace9fdb7f64ecbcff992a"
server_key_iv: "595c160b676a72614927ce4b8d56d111"
server_key_version: "encryptionkey@0"
I20260928 14:56:46.748165 4215 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 14:56:46.752673 4229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:46.754248 4215 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:56:46.754443 4215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/ts-0/wal
uuid: "fcf59744cadd4bb989354022b1ea1756"
format_stamp: "Formatted at 2026-09-28 14:56:46 on dist-test-slave-j968"
server_key: "c91f972cd02ace9fdb7f64ecbcff992a"
server_key_iv: "595c160b676a72614927ce4b8d56d111"
server_key_version: "encryptionkey@0"
I20260928 14:56:46.754689 4215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:56:46.760962 4215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:56:46.761843 4215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:56:46.762156 4215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:56:46.763206 4215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:56:46.765106 4215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:56:46.765199 4215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:46.765286 4215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:56:46.765342 4215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:46.800954 4215 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.1:36833
I20260928 14:56:46.801157 4341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.1:36833 every 8 connection(s)
I20260928 14:56:46.802443 4215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
I20260928 14:56:46.809788 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 4215
I20260928 14:56:46.809998 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/ts-0/wal/instance
I20260928 14:56:46.810806 31904 external_mini_cluster.cc:1428] Setting key e335bd06fa00e4b5f1554ec696d5b300
I20260928 14:56:46.811997 4344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:46.805267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:46399 (local address 127.31.40.62:35669)
0928 14:56:46.805767 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:46.805771 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:46.807369 (+ 1598us) server_negotiation.cc:407] Connection header received
0928 14:56:46.809161 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:46.809174 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:46.809262 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:46.809407 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:46.811121 (+ 1714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:46.811128 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:46.811132 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:56:46.811153 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:46.811203 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:56:46.811220 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:46.811354 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:46.811586 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 14:56:46.811640 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":299,"threads_started":1}
I20260928 14:56:46.813345 4345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:46.805996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:35669 (local address 127.31.40.1:46399)
0928 14:56:46.807214 (+ 1218us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:46.807253 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:46.808866 (+ 1613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:46.809654 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:46.809673 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:46.809788 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:56:46.810693 (+ 905us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:56:46.810696 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:56:46.810739 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:56:46.810837 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:56:46.811329 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:56:46.811338 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:56:46.811343 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:56:46.811684 (+ 341us) client_negotiation.cc:240] Negotiation successful
0928 14:56:46.812118 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":301,"threads_started":1}
I20260928 14:56:46.813866 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.2:0
--local_ip_for_outbound_sockets=127.31.40.2
--webserver_interface=127.31.40.2
--webserver_port=0
--tserver_master_addrs=127.31.40.62:35669
--builtin_ntp_servers=127.31.40.20:32913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 14:56:46.814996 4342 heartbeater.cc:347] Connected to a master server at 127.31.40.62:35669
I20260928 14:56:46.815321 4342 heartbeater.cc:464] Registering TS with master...
I20260928 14:56:46.816071 4342 heartbeater.cc:511] Master 127.31.40.62:35669 requested a full tablet report, sending...
I20260928 14:56:46.818581 4156 ts_manager.cc:194] Registered new tserver with Master: fcf59744cadd4bb989354022b1ea1756 (127.31.40.1:36833)
W20260928 14:56:47.152613 4346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:56:47.152910 4346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:56:47.153003 4346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:56:47.160817 4346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:56:47.160930 4346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:56:47.161042 4346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.2
I20260928 14:56:47.170462 4346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:32913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.40.2
--webserver_port=0
--tserver_master_addrs=127.31.40.62:35669
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.2
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:56:47.171788 4346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:56:47.173661 4346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:56:47.187568 4351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:47.187994 4352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:47.188271 4354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:47.193620 4346 server_base.cc:1034] running on GCE node
I20260928 14:56:47.194303 4346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:56:47.195390 4346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:56:47.196590 4346 hybrid_clock.cc:648] HybridClock initialized: now 1790607407196539 us; error 43 us; skew 500 ppm
I20260928 14:56:47.196939 4346 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 14:56:47.199440 4346 webserver.cc:460] Webserver started at http://127.31.40.2:42769/ using document root <none> and password file <none>
I20260928 14:56:47.200263 4346 fs_manager.cc:359] Metadata directory not provided
I20260928 14:56:47.200394 4346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:56:47.200744 4346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:56:47.203708 4346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/ts-1/data/instance:
uuid: "660338b1e1aa4c44ab9eb55c82401595"
format_stamp: "Formatted at 2026-09-28 14:56:47 on dist-test-slave-j968"
server_key: "73a9ac222125eb72188cd84632ea67cb"
server_key_iv: "f940ceb6bb42b4179aee5bf4bb120cec"
server_key_version: "encryptionkey@0"
I20260928 14:56:47.204653 4346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/ts-1/wal/instance:
uuid: "660338b1e1aa4c44ab9eb55c82401595"
format_stamp: "Formatted at 2026-09-28 14:56:47 on dist-test-slave-j968"
server_key: "73a9ac222125eb72188cd84632ea67cb"
server_key_iv: "f940ceb6bb42b4179aee5bf4bb120cec"
server_key_version: "encryptionkey@0"
I20260928 14:56:47.210404 4346 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 14:56:47.214269 4360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:47.215806 4346 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 14:56:47.216019 4346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/ts-1/wal
uuid: "660338b1e1aa4c44ab9eb55c82401595"
format_stamp: "Formatted at 2026-09-28 14:56:47 on dist-test-slave-j968"
server_key: "73a9ac222125eb72188cd84632ea67cb"
server_key_iv: "f940ceb6bb42b4179aee5bf4bb120cec"
server_key_version: "encryptionkey@0"
I20260928 14:56:47.216305 4346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:56:47.222456 4346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:56:47.223346 4346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:56:47.223640 4346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:56:47.224709 4346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:56:47.226221 4346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:56:47.226310 4346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:47.226475 4346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:56:47.226573 4346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:47.261687 4346 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.2:33833
I20260928 14:56:47.261785 4472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.2:33833 every 8 connection(s)
I20260928 14:56:47.263079 4346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
I20260928 14:56:47.267774 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 4346
I20260928 14:56:47.267978 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/ts-1/wal/instance
I20260928 14:56:47.268440 31904 external_mini_cluster.cc:1428] Setting key 598386080b0fc15832a6f26c18c04de1
I20260928 14:56:47.271310 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.3:0
--local_ip_for_outbound_sockets=127.31.40.3
--webserver_interface=127.31.40.3
--webserver_port=0
--tserver_master_addrs=127.31.40.62:35669
--builtin_ntp_servers=127.31.40.20:32913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 14:56:47.272166 4344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:47.265986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:36845 (local address 127.31.40.62:35669)
0928 14:56:47.266167 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:47.266171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:47.267601 (+ 1430us) server_negotiation.cc:407] Connection header received
0928 14:56:47.269612 (+ 2011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:47.269617 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:47.269684 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:47.269804 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:47.271311 (+ 1507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:47.271317 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:47.271322 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:56:47.271340 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:47.271385 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:56:47.271397 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:47.271532 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:47.271799 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 14:56:47.271889 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 14:56:47.273171 4475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:47.266508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:35669 (local address 127.31.40.2:36845)
0928 14:56:47.267421 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:47.267459 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:47.269338 (+ 1879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:47.270057 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:47.270065 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:47.270205 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:56:47.270998 (+ 793us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:56:47.271001 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:56:47.271040 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:56:47.271128 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:56:47.271595 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:56:47.271603 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:56:47.271609 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 14:56:47.271901 (+ 292us) client_negotiation.cc:240] Negotiation successful
0928 14:56:47.272172 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":277,"threads_started":1}
I20260928 14:56:47.274662 4473 heartbeater.cc:347] Connected to a master server at 127.31.40.62:35669
I20260928 14:56:47.275003 4473 heartbeater.cc:464] Registering TS with master...
I20260928 14:56:47.275637 4473 heartbeater.cc:511] Master 127.31.40.62:35669 requested a full tablet report, sending...
I20260928 14:56:47.277244 4156 ts_manager.cc:194] Registered new tserver with Master: 660338b1e1aa4c44ab9eb55c82401595 (127.31.40.2:33833)
W20260928 14:56:47.614681 4476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:56:47.614995 4476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:56:47.615085 4476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:56:47.622689 4476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:56:47.622809 4476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:56:47.622929 4476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.3
I20260928 14:56:47.632766 4476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:32913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.40.3
--webserver_port=0
--tserver_master_addrs=127.31.40.62:35669
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.3
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:56:47.634145 4476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:56:47.636161 4476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:56:47.650795 4484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:47.650905 4481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:47.651150 4482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:47.652546 4476 server_base.cc:1034] running on GCE node
I20260928 14:56:47.653287 4476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:56:47.654412 4476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:56:47.655642 4476 hybrid_clock.cc:648] HybridClock initialized: now 1790607407655560 us; error 74 us; skew 500 ppm
I20260928 14:56:47.656118 4476 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 14:56:47.658591 4476 webserver.cc:460] Webserver started at http://127.31.40.3:33967/ using document root <none> and password file <none>
I20260928 14:56:47.659487 4476 fs_manager.cc:359] Metadata directory not provided
I20260928 14:56:47.659579 4476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:56:47.660046 4476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:56:47.663057 4476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/ts-2/data/instance:
uuid: "75ad460d1c8740e4877bc5d1c93614d9"
format_stamp: "Formatted at 2026-09-28 14:56:47 on dist-test-slave-j968"
server_key: "a7f3b71379e10cdcaca10792beed6c1c"
server_key_iv: "303e0e5e479b36e985069ce10e83c772"
server_key_version: "encryptionkey@0"
I20260928 14:56:47.663945 4476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/ts-2/wal/instance:
uuid: "75ad460d1c8740e4877bc5d1c93614d9"
format_stamp: "Formatted at 2026-09-28 14:56:47 on dist-test-slave-j968"
server_key: "a7f3b71379e10cdcaca10792beed6c1c"
server_key_iv: "303e0e5e479b36e985069ce10e83c772"
server_key_version: "encryptionkey@0"
I20260928 14:56:47.669726 4476 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 14:56:47.673767 4490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:47.675551 4476 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 14:56:47.675774 4476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/ts-2/wal
uuid: "75ad460d1c8740e4877bc5d1c93614d9"
format_stamp: "Formatted at 2026-09-28 14:56:47 on dist-test-slave-j968"
server_key: "a7f3b71379e10cdcaca10792beed6c1c"
server_key_iv: "303e0e5e479b36e985069ce10e83c772"
server_key_version: "encryptionkey@0"
I20260928 14:56:47.676108 4476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:56:47.683272 4476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:56:47.684237 4476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:56:47.684604 4476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:56:47.685662 4476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:56:47.687235 4476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:56:47.687361 4476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:47.687505 4476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:56:47.687594 4476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:47.723340 4476 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.3:44737
I20260928 14:56:47.723577 4603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.3:44737 every 8 connection(s)
I20260928 14:56:47.724858 4476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
I20260928 14:56:47.734045 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 4476
I20260928 14:56:47.734135 4344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:47.727995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:59577 (local address 127.31.40.62:35669)
0928 14:56:47.728186 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:47.728191 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:47.729761 (+ 1570us) server_negotiation.cc:407] Connection header received
0928 14:56:47.731514 (+ 1753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:47.731520 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:47.731579 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:47.731688 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:47.733140 (+ 1452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:47.733145 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:47.733149 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:56:47.733168 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:47.733203 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:56:47.733215 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:47.733328 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:47.733617 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 14:56:47.733701 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 14:56:47.734229 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/ts-2/wal/instance
I20260928 14:56:47.734714 31904 external_mini_cluster.cc:1428] Setting key 8dd99d3953cb26f6868b2db894c74636
I20260928 14:56:47.734982 4606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:47.728552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:35669 (local address 127.31.40.3:59577)
0928 14:56:47.729594 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:47.729638 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:47.731167 (+ 1529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:47.732001 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:47.732011 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:47.732115 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:56:47.732793 (+ 678us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:56:47.732796 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:56:47.732839 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:56:47.732933 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:56:47.733309 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:56:47.733316 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:56:47.733321 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:56:47.733566 (+ 245us) client_negotiation.cc:240] Negotiation successful
0928 14:56:47.733789 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":298,"threads_started":1}
I20260928 14:56:47.736742 4604 heartbeater.cc:347] Connected to a master server at 127.31.40.62:35669
I20260928 14:56:47.737108 4604 heartbeater.cc:464] Registering TS with master...
I20260928 14:56:47.737892 4604 heartbeater.cc:511] Master 127.31.40.62:35669 requested a full tablet report, sending...
I20260928 14:56:47.739475 4156 ts_manager.cc:194] Registered new tserver with Master: 75ad460d1c8740e4877bc5d1c93614d9 (127.31.40.3:44737)
I20260928 14:56:47.751454 31904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:56:47.759306 4344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:47.756264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50048 (local address 127.31.40.62:35669)
0928 14:56:47.756544 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:47.756552 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:47.757011 (+ 459us) server_negotiation.cc:407] Connection header received
0928 14:56:47.757257 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:47.757262 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:47.757319 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:47.757444 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:47.758629 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:47.758634 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:47.758638 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:56:47.758656 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:47.758689 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:56:47.758702 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:47.758834 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:47.758936 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 14:56:47.759002 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 14:56:47.763674 4156 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50048:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:56:47.771216 4156 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:56:47.793385 4617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:47.786783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:36833 (local address 127.0.0.1:54540)
0928 14:56:47.790004 (+ 3221us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:47.790023 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:47.790149 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:47.790865 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:47.790875 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:47.791016 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:56:47.791942 (+ 926us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:56:47.791945 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:56:47.791974 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:56:47.792057 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:56:47.792562 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:56:47.792575 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:56:47.792584 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 14:56:47.792796 (+ 212us) client_negotiation.cc:240] Negotiation successful
0928 14:56:47.792916 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3099,"thread_start_us":2967,"threads_started":1}
I20260928 14:56:47.793959 4616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:47.786681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.2:33833 (local address 127.0.0.1:44276)
0928 14:56:47.787399 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:47.787449 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:47.787586 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:47.791981 (+ 4395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:47.791986 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:47.792019 (+ 33us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:56:47.792717 (+ 698us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:56:47.792722 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:56:47.792739 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:56:47.792792 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:56:47.793284 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:56:47.793290 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:56:47.793299 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 14:56:47.793449 (+ 150us) client_negotiation.cc:240] Negotiation successful
0928 14:56:47.793505 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":408,"threads_started":1}
I20260928 14:56:47.794427 4619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:47.787734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54540 (local address 127.31.40.1:36833)
0928 14:56:47.788230 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:47.788236 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:47.790135 (+ 1899us) server_negotiation.cc:407] Connection header received
0928 14:56:47.790307 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:47.790320 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:47.790642 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:47.790807 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:47.792263 (+ 1456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:47.792276 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:47.792298 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:56:47.792324 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:47.792397 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:56:47.792420 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:47.793882 (+ 1462us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:47.793983 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 14:56:47.794116 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":256,"threads_started":1}
I20260928 14:56:47.794711 4618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:47.787297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44276 (local address 127.31.40.2:33833)
0928 14:56:47.790639 (+ 3342us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:47.790648 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:47.790676 (+ 28us) server_negotiation.cc:407] Connection header received
0928 14:56:47.790746 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:47.790756 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:47.790979 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:47.791116 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:47.792976 (+ 1860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:47.792985 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:47.792998 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:56:47.793027 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:47.793084 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:56:47.793102 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:47.793223 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:47.794271 (+ 1048us) server_negotiation.cc:299] Negotiation successful
0928 14:56:47.794382 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3189,"thread_start_us":3066,"threads_started":1}
I20260928 14:56:47.796394 4621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:47.792069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.3:44737 (local address 127.0.0.1:55358)
0928 14:56:47.792708 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:47.792731 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:47.792838 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:47.794524 (+ 1686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:47.794529 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:47.794555 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:56:47.795176 (+ 621us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:56:47.795178 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:56:47.795191 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:56:47.795225 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:56:47.795670 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:56:47.795677 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:56:47.795682 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:56:47.795847 (+ 165us) client_negotiation.cc:240] Negotiation successful
0928 14:56:47.795995 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":529,"threads_started":1}
I20260928 14:56:47.797159 4620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:47.790578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55358 (local address 127.31.40.3:44737)
0928 14:56:47.791980 (+ 1402us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:47.791989 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:47.792950 (+ 961us) server_negotiation.cc:407] Connection header received
0928 14:56:47.793028 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:47.793036 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:47.793249 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:47.793399 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:47.795395 (+ 1996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:47.795406 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:47.795420 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:56:47.795440 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:47.795502 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:56:47.795523 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:47.795712 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:47.795829 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 14:56:47.795963 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1191,"thread_start_us":1173,"threads_started":1}
I20260928 14:56:47.798883 4277 tablet_service.cc:1467] Processing CreateTablet for tablet c572b4efc50a4fa68f79964af8c43e6d (DEFAULT_TABLE table=test-table [id=ddee09fbf2254e59ac39def7ee4a55fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:56:47.799156 4408 tablet_service.cc:1467] Processing CreateTablet for tablet c572b4efc50a4fa68f79964af8c43e6d (DEFAULT_TABLE table=test-table [id=ddee09fbf2254e59ac39def7ee4a55fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:56:47.800580 4408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c572b4efc50a4fa68f79964af8c43e6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:56:47.800630 4277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c572b4efc50a4fa68f79964af8c43e6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:56:47.801463 4538 tablet_service.cc:1467] Processing CreateTablet for tablet c572b4efc50a4fa68f79964af8c43e6d (DEFAULT_TABLE table=test-table [id=ddee09fbf2254e59ac39def7ee4a55fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:56:47.803120 4538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c572b4efc50a4fa68f79964af8c43e6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:56:47.814311 4622 tablet_bootstrap.cc:492] T c572b4efc50a4fa68f79964af8c43e6d P 660338b1e1aa4c44ab9eb55c82401595: Bootstrap starting.
I20260928 14:56:47.818697 4623 tablet_bootstrap.cc:492] T c572b4efc50a4fa68f79964af8c43e6d P 75ad460d1c8740e4877bc5d1c93614d9: Bootstrap starting.
I20260928 14:56:47.818780 4622 tablet_bootstrap.cc:654] T c572b4efc50a4fa68f79964af8c43e6d P 660338b1e1aa4c44ab9eb55c82401595: Neither blocks nor log segments found. Creating new log.
I20260928 14:56:47.820557 4622 log.cc:824] T c572b4efc50a4fa68f79964af8c43e6d P 660338b1e1aa4c44ab9eb55c82401595: Log is configured to *not* fsync() on all Append() calls
I20260928 14:56:47.822017 4342 heartbeater.cc:503] Master 127.31.40.62:35669 was elected leader, sending a full tablet report...
I20260928 14:56:47.823760 4623 tablet_bootstrap.cc:654] T c572b4efc50a4fa68f79964af8c43e6d P 75ad460d1c8740e4877bc5d1c93614d9: Neither blocks nor log segments found. Creating new log.
I20260928 14:56:47.824179 4622 tablet_bootstrap.cc:492] T c572b4efc50a4fa68f79964af8c43e6d P 660338b1e1aa4c44ab9eb55c82401595: No bootstrap required, opened a new log
I20260928 14:56:47.824492 4622 ts_tablet_manager.cc:1397] T c572b4efc50a4fa68f79964af8c43e6d P 660338b1e1aa4c44ab9eb55c82401595: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260928 14:56:47.825155 4623 log.cc:824] T c572b4efc50a4fa68f79964af8c43e6d P 75ad460d1c8740e4877bc5d1c93614d9: Log is configured to *not* fsync() on all Append() calls
I20260928 14:56:47.830847 4623 tablet_bootstrap.cc:492] T c572b4efc50a4fa68f79964af8c43e6d P 75ad460d1c8740e4877bc5d1c93614d9: No bootstrap required, opened a new log
I20260928 14:56:47.831182 4623 ts_tablet_manager.cc:1397] T c572b4efc50a4fa68f79964af8c43e6d P 75ad460d1c8740e4877bc5d1c93614d9: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260928 14:56:47.831234 4625 tablet_bootstrap.cc:492] T c572b4efc50a4fa68f79964af8c43e6d P fcf59744cadd4bb989354022b1ea1756: Bootstrap starting.
I20260928 14:56:47.836272 4625 tablet_bootstrap.cc:654] T c572b4efc50a4fa68f79964af8c43e6d P fcf59744cadd4bb989354022b1ea1756: Neither blocks nor log segments found. Creating new log.
I20260928 14:56:47.837204 4622 raft_consensus.cc:348] T c572b4efc50a4fa68f79964af8c43e6d P 660338b1e1aa4c44ab9eb55c82401595 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "660338b1e1aa4c44ab9eb55c82401595" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 33833 } } peers { permanent_uuid: "fcf59744cadd4bb989354022b1ea1756" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36833 } } peers { permanent_uuid: "75ad460d1c8740e4877bc5d1c93614d9" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 44737 } }
I20260928 14:56:47.837764 4622 raft_consensus.cc:374] T c572b4efc50a4fa68f79964af8c43e6d P 660338b1e1aa4c44ab9eb55c82401595 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:56:47.837883 4622 raft_consensus.cc:729] T c572b4efc50a4fa68f79964af8c43e6d P 660338b1e1aa4c44ab9eb55c82401595 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 660338b1e1aa4c44ab9eb55c82401595, State: Initialized, Role: FOLLOWER
I20260928 14:56:47.837927 4625 log.cc:824] T c572b4efc50a4fa68f79964af8c43e6d P fcf59744cadd4bb989354022b1ea1756: Log is configured to *not* fsync() on all Append() calls
I20260928 14:56:47.838708 4622 consensus_queue.cc:260] T c572b4efc50a4fa68f79964af8c43e6d P 660338b1e1aa4c44ab9eb55c82401595 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "660338b1e1aa4c44ab9eb55c82401595" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 33833 } } peers { permanent_uuid: "fcf59744cadd4bb989354022b1ea1756" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36833 } } peers { permanent_uuid: "75ad460d1c8740e4877bc5d1c93614d9" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 44737 } }
I20260928 14:56:47.840617 4473 heartbeater.cc:503] Master 127.31.40.62:35669 was elected leader, sending a full tablet report...
I20260928 14:56:47.841483 4622 ts_tablet_manager.cc:1428] T c572b4efc50a4fa68f79964af8c43e6d P 660338b1e1aa4c44ab9eb55c82401595: Time spent starting tablet: real 0.017s user 0.015s sys 0.000s
I20260928 14:56:47.841907 4625 tablet_bootstrap.cc:492] T c572b4efc50a4fa68f79964af8c43e6d P fcf59744cadd4bb989354022b1ea1756: No bootstrap required, opened a new log
I20260928 14:56:47.842249 4625 ts_tablet_manager.cc:1397] T c572b4efc50a4fa68f79964af8c43e6d P fcf59744cadd4bb989354022b1ea1756: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.004s
I20260928 14:56:47.842640 4623 raft_consensus.cc:348] T c572b4efc50a4fa68f79964af8c43e6d P 75ad460d1c8740e4877bc5d1c93614d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "660338b1e1aa4c44ab9eb55c82401595" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 33833 } } peers { permanent_uuid: "fcf59744cadd4bb989354022b1ea1756" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36833 } } peers { permanent_uuid: "75ad460d1c8740e4877bc5d1c93614d9" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 44737 } }
I20260928 14:56:47.843045 4623 raft_consensus.cc:374] T c572b4efc50a4fa68f79964af8c43e6d P 75ad460d1c8740e4877bc5d1c93614d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:56:47.843164 4623 raft_consensus.cc:729] T c572b4efc50a4fa68f79964af8c43e6d P 75ad460d1c8740e4877bc5d1c93614d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75ad460d1c8740e4877bc5d1c93614d9, State: Initialized, Role: FOLLOWER
I20260928 14:56:47.844106 4623 consensus_queue.cc:260] T c572b4efc50a4fa68f79964af8c43e6d P 75ad460d1c8740e4877bc5d1c93614d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "660338b1e1aa4c44ab9eb55c82401595" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 33833 } } peers { permanent_uuid: "fcf59744cadd4bb989354022b1ea1756" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36833 } } peers { permanent_uuid: "75ad460d1c8740e4877bc5d1c93614d9" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 44737 } }
I20260928 14:56:47.846040 4623 ts_tablet_manager.cc:1428] T c572b4efc50a4fa68f79964af8c43e6d P 75ad460d1c8740e4877bc5d1c93614d9: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260928 14:56:47.846498 4604 heartbeater.cc:503] Master 127.31.40.62:35669 was elected leader, sending a full tablet report...
I20260928 14:56:47.852289 4625 raft_consensus.cc:348] T c572b4efc50a4fa68f79964af8c43e6d P fcf59744cadd4bb989354022b1ea1756 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "660338b1e1aa4c44ab9eb55c82401595" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 33833 } } peers { permanent_uuid: "fcf59744cadd4bb989354022b1ea1756" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36833 } } peers { permanent_uuid: "75ad460d1c8740e4877bc5d1c93614d9" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 44737 } }
I20260928 14:56:47.852768 4625 raft_consensus.cc:374] T c572b4efc50a4fa68f79964af8c43e6d P fcf59744cadd4bb989354022b1ea1756 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:56:47.852880 4625 raft_consensus.cc:729] T c572b4efc50a4fa68f79964af8c43e6d P fcf59744cadd4bb989354022b1ea1756 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcf59744cadd4bb989354022b1ea1756, State: Initialized, Role: FOLLOWER
I20260928 14:56:47.853520 4625 consensus_queue.cc:260] T c572b4efc50a4fa68f79964af8c43e6d P fcf59744cadd4bb989354022b1ea1756 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "660338b1e1aa4c44ab9eb55c82401595" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 33833 } } peers { permanent_uuid: "fcf59744cadd4bb989354022b1ea1756" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36833 } } peers { permanent_uuid: "75ad460d1c8740e4877bc5d1c93614d9" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 44737 } }
I20260928 14:56:47.855298 4625 ts_tablet_manager.cc:1428] T c572b4efc50a4fa68f79964af8c43e6d P fcf59744cadd4bb989354022b1ea1756: Time spent starting tablet: real 0.013s user 0.007s sys 0.007s
W20260928 14:56:47.977180 4605 tablet.cc:2369] T c572b4efc50a4fa68f79964af8c43e6d P 75ad460d1c8740e4877bc5d1c93614d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:56:48.015473 4474 tablet.cc:2369] T c572b4efc50a4fa68f79964af8c43e6d P 660338b1e1aa4c44ab9eb55c82401595: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:56:48.055034 4343 tablet.cc:2369] T c572b4efc50a4fa68f79964af8c43e6d P fcf59744cadd4bb989354022b1ea1756: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:56:48.203060 4630 raft_consensus.cc:482] T c572b4efc50a4fa68f79964af8c43e6d P fcf59744cadd4bb989354022b1ea1756 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:56:48.203451 4630 raft_consensus.cc:504] T c572b4efc50a4fa68f79964af8c43e6d P fcf59744cadd4bb989354022b1ea1756 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "660338b1e1aa4c44ab9eb55c82401595" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 33833 } } peers { permanent_uuid: "fcf59744cadd4bb989354022b1ea1756" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36833 } } peers { permanent_uuid: "75ad460d1c8740e4877bc5d1c93614d9" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 44737 } }
I20260928 14:56:48.205410 4630 leader_election.cc:288] T c572b4efc50a4fa68f79964af8c43e6d P fcf59744cadd4bb989354022b1ea1756 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 660338b1e1aa4c44ab9eb55c82401595 (127.31.40.2:33833), 75ad460d1c8740e4877bc5d1c93614d9 (127.31.40.3:44737)
I20260928 14:56:48.209717 4620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:48.206073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:48919 (local address 127.31.40.3:44737)
0928 14:56:48.206309 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:48.206317 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:48.207623 (+ 1306us) server_negotiation.cc:407] Connection header received
0928 14:56:48.207730 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:48.207736 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:48.207796 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:48.207921 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:48.209032 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:48.209036 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:48.209039 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:56:48.209056 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:48.209086 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:56:48.209119 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:48.209249 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:48.209385 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 14:56:48.209459 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 14:56:48.209721 4618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:48.205898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:33771 (local address 127.31.40.2:33833)
0928 14:56:48.206163 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:48.206172 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:48.207108 (+ 936us) server_negotiation.cc:407] Connection header received
0928 14:56:48.207351 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:48.207358 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:48.207427 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:48.207509 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:48.208957 (+ 1448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:48.208964 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:48.208970 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:56:48.208990 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:48.209031 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:56:48.209047 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:48.209165 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:48.209350 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 14:56:48.209421 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260928 14:56:48.209847 4632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:48.205798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.2:33833 (local address 127.31.40.1:33771)
0928 14:56:48.206980 (+ 1182us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:48.206997 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:48.207163 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:48.207718 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:48.207723 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:48.207761 (+ 38us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:56:48.208686 (+ 925us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:56:48.208692 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:56:48.208713 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:56:48.208776 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:56:48.209154 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:56:48.209162 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:56:48.209171 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 14:56:48.209365 (+ 194us) client_negotiation.cc:240] Negotiation successful
0928 14:56:48.209420 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1045,"thread_start_us":830,"threads_started":1}
I20260928 14:56:48.210464 4631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:48.205812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.3:44737 (local address 127.31.40.1:48919)
0928 14:56:48.207496 (+ 1684us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:48.207513 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:48.207619 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:48.207978 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:48.207982 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:48.208007 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:56:48.208805 (+ 798us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:56:48.208810 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:56:48.208826 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:56:48.208867 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:56:48.209234 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:56:48.209239 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:56:48.209245 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 14:56:48.209962 (+ 717us) client_negotiation.cc:240] Negotiation successful
0928 14:56:48.210013 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1550,"thread_start_us":1312,"threads_started":1}
I20260928 14:56:48.211015 4428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c572b4efc50a4fa68f79964af8c43e6d" candidate_uuid: "fcf59744cadd4bb989354022b1ea1756" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "660338b1e1aa4c44ab9eb55c82401595" is_pre_election: true
I20260928 14:56:48.211585 4428 raft_consensus.cc:2454] T c572b4efc50a4fa68f79964af8c43e6d P 660338b1e1aa4c44ab9eb55c82401595 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcf59744cadd4bb989354022b1ea1756 in term 0.
I20260928 14:56:48.211519 4558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c572b4efc50a4fa68f79964af8c43e6d" candidate_uuid: "fcf59744cadd4bb989354022b1ea1756" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75ad460d1c8740e4877bc5d1c93614d9" is_pre_election: true
I20260928 14:56:48.212019 4558 raft_consensus.cc:2454] T c572b4efc50a4fa68f79964af8c43e6d P 75ad460d1c8740e4877bc5d1c93614d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcf59744cadd4bb989354022b1ea1756 in term 0.
I20260928 14:56:48.212569 4230 leader_election.cc:302] T c572b4efc50a4fa68f79964af8c43e6d P fcf59744cadd4bb989354022b1ea1756 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 660338b1e1aa4c44ab9eb55c82401595, 75ad460d1c8740e4877bc5d1c93614d9, fcf59744cadd4bb989354022b1ea1756; no voters:
I20260928 14:56:48.213165 4630 raft_consensus.cc:2781] T c572b4efc50a4fa68f79964af8c43e6d P fcf59744cadd4bb989354022b1ea1756 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:56:48.213290 4630 raft_consensus.cc:482] T c572b4efc50a4fa68f79964af8c43e6d P fcf59744cadd4bb989354022b1ea1756 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:56:48.213413 4630 raft_consensus.cc:3037] T c572b4efc50a4fa68f79964af8c43e6d P fcf59744cadd4bb989354022b1ea1756 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:56:48.215636 4630 raft_consensus.cc:504] T c572b4efc50a4fa68f79964af8c43e6d P fcf59744cadd4bb989354022b1ea1756 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "660338b1e1aa4c44ab9eb55c82401595" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 33833 } } peers { permanent_uuid: "fcf59744cadd4bb989354022b1ea1756" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36833 } } peers { permanent_uuid: "75ad460d1c8740e4877bc5d1c93614d9" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 44737 } }
I20260928 14:56:48.216378 4630 leader_election.cc:288] T c572b4efc50a4fa68f79964af8c43e6d P fcf59744cadd4bb989354022b1ea1756 [CANDIDATE]: Term 1 election: Requested vote from peers 660338b1e1aa4c44ab9eb55c82401595 (127.31.40.2:33833), 75ad460d1c8740e4877bc5d1c93614d9 (127.31.40.3:44737)
I20260928 14:56:48.216958 4428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c572b4efc50a4fa68f79964af8c43e6d" candidate_uuid: "fcf59744cadd4bb989354022b1ea1756" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "660338b1e1aa4c44ab9eb55c82401595"
I20260928 14:56:48.216941 4558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c572b4efc50a4fa68f79964af8c43e6d" candidate_uuid: "fcf59744cadd4bb989354022b1ea1756" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75ad460d1c8740e4877bc5d1c93614d9"
I20260928 14:56:48.217182 4428 raft_consensus.cc:3037] T c572b4efc50a4fa68f79964af8c43e6d P 660338b1e1aa4c44ab9eb55c82401595 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:56:48.217182 4558 raft_consensus.cc:3037] T c572b4efc50a4fa68f79964af8c43e6d P 75ad460d1c8740e4877bc5d1c93614d9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:56:48.220896 4428 raft_consensus.cc:2454] T c572b4efc50a4fa68f79964af8c43e6d P 660338b1e1aa4c44ab9eb55c82401595 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcf59744cadd4bb989354022b1ea1756 in term 1.
I20260928 14:56:48.220904 4558 raft_consensus.cc:2454] T c572b4efc50a4fa68f79964af8c43e6d P 75ad460d1c8740e4877bc5d1c93614d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcf59744cadd4bb989354022b1ea1756 in term 1.
I20260928 14:56:48.221573 4232 leader_election.cc:302] T c572b4efc50a4fa68f79964af8c43e6d P fcf59744cadd4bb989354022b1ea1756 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75ad460d1c8740e4877bc5d1c93614d9, fcf59744cadd4bb989354022b1ea1756; no voters:
I20260928 14:56:48.222054 4630 raft_consensus.cc:2781] T c572b4efc50a4fa68f79964af8c43e6d P fcf59744cadd4bb989354022b1ea1756 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:56:48.222586 4630 raft_consensus.cc:686] T c572b4efc50a4fa68f79964af8c43e6d P fcf59744cadd4bb989354022b1ea1756 [term 1 LEADER]: Becoming Leader. State: Replica: fcf59744cadd4bb989354022b1ea1756, State: Running, Role: LEADER
I20260928 14:56:48.223209 4630 consensus_queue.cc:237] T c572b4efc50a4fa68f79964af8c43e6d P fcf59744cadd4bb989354022b1ea1756 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "660338b1e1aa4c44ab9eb55c82401595" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 33833 } } peers { permanent_uuid: "fcf59744cadd4bb989354022b1ea1756" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36833 } } peers { permanent_uuid: "75ad460d1c8740e4877bc5d1c93614d9" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 44737 } }
I20260928 14:56:48.228572 4155 catalog_manager.cc:5495] T c572b4efc50a4fa68f79964af8c43e6d P fcf59744cadd4bb989354022b1ea1756 reported cstate change: term changed from 0 to 1, leader changed from <none> to fcf59744cadd4bb989354022b1ea1756 (127.31.40.1). New cstate: current_term: 1 leader_uuid: "fcf59744cadd4bb989354022b1ea1756" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "660338b1e1aa4c44ab9eb55c82401595" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 33833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fcf59744cadd4bb989354022b1ea1756" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75ad460d1c8740e4877bc5d1c93614d9" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 44737 } health_report { overall_health: UNKNOWN } } }
I20260928 14:56:48.263918 4619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:48.260795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54544 (local address 127.31.40.1:36833)
0928 14:56:48.261085 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:48.261095 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:48.261650 (+ 555us) server_negotiation.cc:407] Connection header received
0928 14:56:48.261870 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:48.261876 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:48.261997 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:48.262119 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:48.263233 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:48.263239 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:48.263243 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:56:48.263262 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:48.263302 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:56:48.263318 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:48.263452 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:48.263554 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 14:56:48.263617 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 14:56:48.273451 4428 raft_consensus.cc:1261] T c572b4efc50a4fa68f79964af8c43e6d P 660338b1e1aa4c44ab9eb55c82401595 [term 1 FOLLOWER]: Refusing update from remote peer fcf59744cadd4bb989354022b1ea1756: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:56:48.273447 4558 raft_consensus.cc:1261] T c572b4efc50a4fa68f79964af8c43e6d P 75ad460d1c8740e4877bc5d1c93614d9 [term 1 FOLLOWER]: Refusing update from remote peer fcf59744cadd4bb989354022b1ea1756: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:56:48.274609 4630 consensus_queue.cc:1035] T c572b4efc50a4fa68f79964af8c43e6d P fcf59744cadd4bb989354022b1ea1756 [LEADER]: Connected to new peer: Peer: permanent_uuid: "660338b1e1aa4c44ab9eb55c82401595" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 33833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:56:48.274961 4633 consensus_queue.cc:1035] T c572b4efc50a4fa68f79964af8c43e6d P fcf59744cadd4bb989354022b1ea1756 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75ad460d1c8740e4877bc5d1c93614d9" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 44737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:56:48.287374 4637 mvcc.cc:205] Tried to move back new op lower bound from 7334327944275972096 to 7334327944087015424. Current Snapshot: MvccSnapshot[applied={T|T < 7334327944275972096}]
I20260928 14:56:48.292999 4638 mvcc.cc:205] Tried to move back new op lower bound from 7334327944275972096 to 7334327944087015424. Current Snapshot: MvccSnapshot[applied={T|T < 7334327944275972096}]
I20260928 14:56:48.304693 4155 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:56:48.305109 4155 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:56:48.309545 4155 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a7a9429a791946db92674624cd83b760: Sending DeleteTablet for 3 replicas of tablet c572b4efc50a4fa68f79964af8c43e6d
I20260928 14:56:48.311080 4408 tablet_service.cc:1514] Processing DeleteTablet for tablet c572b4efc50a4fa68f79964af8c43e6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:56:48 UTC) from {username='slave'} at 127.0.0.1:44276
I20260928 14:56:48.311144 4538 tablet_service.cc:1514] Processing DeleteTablet for tablet c572b4efc50a4fa68f79964af8c43e6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:56:48 UTC) from {username='slave'} at 127.0.0.1:55358
I20260928 14:56:48.311427 4277 tablet_service.cc:1514] Processing DeleteTablet for tablet c572b4efc50a4fa68f79964af8c43e6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:56:48 UTC) from {username='slave'} at 127.0.0.1:54540
I20260928 14:56:48.313285 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 4215
I20260928 14:56:48.327385 4650 tablet_replica.cc:330] T c572b4efc50a4fa68f79964af8c43e6d P fcf59744cadd4bb989354022b1ea1756: stopping tablet replica
I20260928 14:56:48.328089 4650 raft_consensus.cc:2229] T c572b4efc50a4fa68f79964af8c43e6d P fcf59744cadd4bb989354022b1ea1756 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:56:48.329011 4650 raft_consensus.cc:2258] T c572b4efc50a4fa68f79964af8c43e6d P fcf59744cadd4bb989354022b1ea1756 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:56:48.332155 4650 ts_tablet_manager.cc:1905] T c572b4efc50a4fa68f79964af8c43e6d P fcf59744cadd4bb989354022b1ea1756: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:56:48.341579 4619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:48.314906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48350 (local address 127.31.40.1:36833)
0928 14:56:48.315180 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:48.315189 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:48.338332 (+ 23143us) server_negotiation.cc:407] Connection header received
0928 14:56:48.339007 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:48.339014 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:48.339089 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:48.339184 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:48.340587 (+ 1403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:48.340593 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:48.340599 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:56:48.340621 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:48.340661 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:56:48.340677 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:48.340791 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:48.341114 (+ 323us) server_negotiation.cc:299] Negotiation successful
0928 14:56:48.341191 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 14:56:48.342571 4650 ts_tablet_manager.cc:1918] T c572b4efc50a4fa68f79964af8c43e6d P fcf59744cadd4bb989354022b1ea1756: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:56:48.342818 4650 log.cc:1196] T c572b4efc50a4fa68f79964af8c43e6d P fcf59744cadd4bb989354022b1ea1756: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/ts-0/wal/wals/c572b4efc50a4fa68f79964af8c43e6d
I20260928 14:56:48.342870 4337 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:56:48.344348 4648 tablet_replica.cc:330] T c572b4efc50a4fa68f79964af8c43e6d P 660338b1e1aa4c44ab9eb55c82401595: stopping tablet replica
I20260928 14:56:48.344877 4649 tablet_replica.cc:330] T c572b4efc50a4fa68f79964af8c43e6d P 75ad460d1c8740e4877bc5d1c93614d9: stopping tablet replica
I20260928 14:56:48.345213 4648 raft_consensus.cc:2229] T c572b4efc50a4fa68f79964af8c43e6d P 660338b1e1aa4c44ab9eb55c82401595 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:56:48.345491 4649 raft_consensus.cc:2229] T c572b4efc50a4fa68f79964af8c43e6d P 75ad460d1c8740e4877bc5d1c93614d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:56:48.345749 4648 raft_consensus.cc:2258] T c572b4efc50a4fa68f79964af8c43e6d P 660338b1e1aa4c44ab9eb55c82401595 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:56:48.346145 4649 raft_consensus.cc:2258] T c572b4efc50a4fa68f79964af8c43e6d P 75ad460d1c8740e4877bc5d1c93614d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:56:48.348721 4648 ts_tablet_manager.cc:1905] T c572b4efc50a4fa68f79964af8c43e6d P 660338b1e1aa4c44ab9eb55c82401595: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:56:48.349056 4649 ts_tablet_manager.cc:1905] T c572b4efc50a4fa68f79964af8c43e6d P 75ad460d1c8740e4877bc5d1c93614d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:56:48.354919 4649 ts_tablet_manager.cc:1918] T c572b4efc50a4fa68f79964af8c43e6d P 75ad460d1c8740e4877bc5d1c93614d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:56:48.355141 4649 log.cc:1196] T c572b4efc50a4fa68f79964af8c43e6d P 75ad460d1c8740e4877bc5d1c93614d9: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/ts-2/wal/wals/c572b4efc50a4fa68f79964af8c43e6d
I20260928 14:56:48.355647 4649 ts_tablet_manager.cc:1939] T c572b4efc50a4fa68f79964af8c43e6d P 75ad460d1c8740e4877bc5d1c93614d9: Deleting consensus metadata
I20260928 14:56:48.357460 4648 ts_tablet_manager.cc:1918] T c572b4efc50a4fa68f79964af8c43e6d P 660338b1e1aa4c44ab9eb55c82401595: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:56:48.357646 4648 log.cc:1196] T c572b4efc50a4fa68f79964af8c43e6d P 660338b1e1aa4c44ab9eb55c82401595: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607365362981-31904-0/minicluster-data/ts-1/wal/wals/c572b4efc50a4fa68f79964af8c43e6d
I20260928 14:56:48.357805 4141 catalog_manager.cc:4841] TS 75ad460d1c8740e4877bc5d1c93614d9 (127.31.40.3:44737): tablet c572b4efc50a4fa68f79964af8c43e6d (table test-table [id=ddee09fbf2254e59ac39def7ee4a55fa]) successfully deleted
I20260928 14:56:48.358273 4648 ts_tablet_manager.cc:1939] T c572b4efc50a4fa68f79964af8c43e6d P 660338b1e1aa4c44ab9eb55c82401595: Deleting consensus metadata
I20260928 14:56:48.360330 4139 catalog_manager.cc:4841] TS 660338b1e1aa4c44ab9eb55c82401595 (127.31.40.2:33833): tablet c572b4efc50a4fa68f79964af8c43e6d (table test-table [id=ddee09fbf2254e59ac39def7ee4a55fa]) successfully deleted
I20260928 14:56:48.449681 4650 ts_tablet_manager.cc:1939] T c572b4efc50a4fa68f79964af8c43e6d P fcf59744cadd4bb989354022b1ea1756: Deleting consensus metadata
I20260928 14:56:48.449812 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 4215
W20260928 14:56:48.468225 4141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.40.1:36833 (error 108)
I20260928 14:56:48.468467 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 4346
I20260928 14:56:48.470427 4654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:48.469549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:36833 (local address 127.0.0.1:48364)
0928 14:56:48.470038 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:48.470180 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.1:36833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":236,"threads_started":1}
W20260928 14:56:48.470670 4141 catalog_manager.cc:4584] TS fcf59744cadd4bb989354022b1ea1756 (127.31.40.1:36833): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c572b4efc50a4fa68f79964af8c43e6d: Network error: Client connection negotiation failed: client connection to 127.31.40.1:36833: connect: Connection refused (error 111)
I20260928 14:56:48.471851 4618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:48.469384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43200 (local address 127.31.40.2:33833)
0928 14:56:48.469567 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:48.469571 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:48.469652 (+ 81us) server_negotiation.cc:407] Connection header received
0928 14:56:48.469902 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:48.469907 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:48.470015 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:48.470129 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:48.471232 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:48.471237 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:48.471248 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:56:48.471263 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:48.471295 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:56:48.471307 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:48.471439 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:48.471529 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 14:56:48.471590 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 14:56:48.472581 4468 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:56:48.524976 4654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:48.524358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:36833 (local address 127.0.0.1:48380)
0928 14:56:48.524657 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:48.524790 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.1:36833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 14:56:48.526163 4654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:48.525724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:36833 (local address 127.0.0.1:48394)
0928 14:56:48.525899 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:48.525990 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.1:36833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 14:56:48.565208 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 4346
I20260928 14:56:48.584014 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 4476
I20260928 14:56:48.586714 4654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:48.586211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:36833 (local address 127.0.0.1:48404)
0928 14:56:48.586416 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:48.586528 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.1:36833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260928 14:56:48.587237 4620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:48.584815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38626 (local address 127.31.40.3:44737)
0928 14:56:48.585037 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:48.585042 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:48.585147 (+ 105us) server_negotiation.cc:407] Connection header received
0928 14:56:48.585312 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:48.585316 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:48.585406 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:48.585522 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:48.586553 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:48.586557 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:48.586560 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:56:48.586575 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:48.586605 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:56:48.586616 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:48.586717 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:48.586873 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 14:56:48.586934 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 14:56:48.587909 4654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:48.587485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:36833 (local address 127.0.0.1:48418)
0928 14:56:48.587666 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:48.587740 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.1:36833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 14:56:48.588305 4599 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:56:48.668496 4654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:48.667959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:36833 (local address 127.0.0.1:48430)
0928 14:56:48.668201 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:48.668307 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.1:36833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260928 14:56:48.669672 4654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:48.669175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:36833 (local address 127.0.0.1:48446)
0928 14:56:48.669402 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:48.669495 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.1:36833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260928 14:56:48.676916 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 4476
I20260928 14:56:48.694088 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 4124
I20260928 14:56:48.695389 4186 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:56:48.805461 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 4124
2026-09-28T14:56:48Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (3098 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 14:56:48.847693 31904 mini_oidc.cc:152] Starting JWKS server
I20260928 14:56:48.850381 31904 webserver.cc:460] Webserver started at https://127.0.0.1:46177/ using document root <none> and password file <none>
I20260928 14:56:48.850770 31904 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 14:56:48.851486 31904 webserver.cc:460] Webserver started at http://127.0.0.1:36263/ using document root <none> and password file <none>
2026-09-28T14:56:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:56:48Z Disabled control of system clock
I20260928 14:56:48.879726 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607365362981-31904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607365362981-31904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607365362981-31904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:42547
--webserver_interface=127.31.40.62
--webserver_port=0
--builtin_ntp_servers=127.31.40.20:32963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.40.62:42547
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46177/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607365362981-31904-0/testchainca.cert with env {}
W20260928 14:56:49.215559 4663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:56:49.215857 4663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:56:49.216039 4663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:56:49.223541 4663 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:56:49.223663 4663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:56:49.223701 4663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:56:49.223752 4663 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:56:49.223795 4663 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 14:56:49.223857 4663 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 14:56:49.223971 4663 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46177/jwks.json
I20260928 14:56:49.234371 4663 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:32963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607365362981-31904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607365362981-31904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.40.62:42547
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:42547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46177/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.40.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607365362981-31904-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607365362981-31904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:56:49.235705 4663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:56:49.237092 4663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:56:49.249651 4671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:49.249594 4669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:49.249594 4668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:49.250633 4663 server_base.cc:1034] running on GCE node
I20260928 14:56:49.251518 4663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:56:49.253057 4663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:56:49.254307 4663 hybrid_clock.cc:648] HybridClock initialized: now 1790607409254259 us; error 37 us; skew 500 ppm
I20260928 14:56:49.254678 4663 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 14:56:49.261106 4663 webserver.cc:460] Webserver started at http://127.31.40.62:38933/ using document root <none> and password file <none>
I20260928 14:56:49.261844 4663 fs_manager.cc:359] Metadata directory not provided
I20260928 14:56:49.262051 4663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:56:49.262378 4663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:56:49.265429 4663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607365362981-31904-0/minicluster-data/master-0/data/instance:
uuid: "b634054633d84ca9bba5a2e3305bc3e8"
format_stamp: "Formatted at 2026-09-28 14:56:49 on dist-test-slave-j968"
server_key: "5a68d31887c586e27fd41a750b9adb24"
server_key_iv: "b4b05dd00739d268f551279419042291"
server_key_version: "encryptionkey@0"
I20260928 14:56:49.266242 4663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607365362981-31904-0/minicluster-data/master-0/wal/instance:
uuid: "b634054633d84ca9bba5a2e3305bc3e8"
format_stamp: "Formatted at 2026-09-28 14:56:49 on dist-test-slave-j968"
server_key: "5a68d31887c586e27fd41a750b9adb24"
server_key_iv: "b4b05dd00739d268f551279419042291"
server_key_version: "encryptionkey@0"
I20260928 14:56:49.271411 4663 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 14:56:49.274950 4677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:49.276296 4663 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:56:49.276480 4663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607365362981-31904-0/minicluster-data/master-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607365362981-31904-0/minicluster-data/master-0/wal
uuid: "b634054633d84ca9bba5a2e3305bc3e8"
format_stamp: "Formatted at 2026-09-28 14:56:49 on dist-test-slave-j968"
server_key: "5a68d31887c586e27fd41a750b9adb24"
server_key_iv: "b4b05dd00739d268f551279419042291"
server_key_version: "encryptionkey@0"
I20260928 14:56:49.276719 4663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607365362981-31904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607365362981-31904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607365362981-31904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:56:49.304188 4663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:56:49.305101 4663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:56:49.305430 4663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:56:49.324837 4663 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.62:42547
I20260928 14:56:49.324975 4728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.62:42547 every 8 connection(s)
I20260928 14:56:49.326247 4663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
I20260928 14:56:49.330091 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 4663
I20260928 14:56:49.330343 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607365362981-31904-0/minicluster-data/master-0/wal/instance
I20260928 14:56:49.330937 4729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:56:49.331231 31904 external_mini_cluster.cc:1428] Setting key 7042f932adefacc855fe305f21b0f10e
I20260928 14:56:49.344944 4729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b634054633d84ca9bba5a2e3305bc3e8: Bootstrap starting.
I20260928 14:56:49.345856 4732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:49.333412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52598 (local address 127.31.40.62:42547)
0928 14:56:49.334583 (+ 1171us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:49.334598 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:49.334645 (+ 47us) server_negotiation.cc:407] Connection header received
0928 14:56:49.336531 (+ 1886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:49.336556 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:49.337084 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:49.337547 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:49.338179 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:49.339569 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:49.341081 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:49.341820 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:49.342088 (+ 268us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:49.343389 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:49.343397 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:49.343415 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:56:49.344040 (+ 625us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:49.344122 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:56:49.344144 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:49.344270 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:49.344520 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 14:56:49.344846 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":441,"threads_started":1}
I20260928 14:56:49.348301 4729 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b634054633d84ca9bba5a2e3305bc3e8: Neither blocks nor log segments found. Creating new log.
I20260928 14:56:49.349447 4729 log.cc:824] T 00000000000000000000000000000000 P b634054633d84ca9bba5a2e3305bc3e8: Log is configured to *not* fsync() on all Append() calls
I20260928 14:56:49.350008 31904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:56:49.352527 4729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b634054633d84ca9bba5a2e3305bc3e8: No bootstrap required, opened a new log
I20260928 14:56:49.360266 4729 raft_consensus.cc:348] T 00000000000000000000000000000000 P b634054633d84ca9bba5a2e3305bc3e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b634054633d84ca9bba5a2e3305bc3e8" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 42547 } }
I20260928 14:56:49.360668 4729 raft_consensus.cc:374] T 00000000000000000000000000000000 P b634054633d84ca9bba5a2e3305bc3e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:56:49.360788 4729 raft_consensus.cc:729] T 00000000000000000000000000000000 P b634054633d84ca9bba5a2e3305bc3e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b634054633d84ca9bba5a2e3305bc3e8, State: Initialized, Role: FOLLOWER
I20260928 14:56:49.361353 4729 consensus_queue.cc:260] T 00000000000000000000000000000000 P b634054633d84ca9bba5a2e3305bc3e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b634054633d84ca9bba5a2e3305bc3e8" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 42547 } }
I20260928 14:56:49.361649 4729 raft_consensus.cc:388] T 00000000000000000000000000000000 P b634054633d84ca9bba5a2e3305bc3e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:56:49.361833 4729 raft_consensus.cc:482] T 00000000000000000000000000000000 P b634054633d84ca9bba5a2e3305bc3e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:56:49.362053 4729 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b634054633d84ca9bba5a2e3305bc3e8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:56:49.365743 4729 raft_consensus.cc:504] T 00000000000000000000000000000000 P b634054633d84ca9bba5a2e3305bc3e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b634054633d84ca9bba5a2e3305bc3e8" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 42547 } }
I20260928 14:56:49.366348 4729 leader_election.cc:302] T 00000000000000000000000000000000 P b634054633d84ca9bba5a2e3305bc3e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b634054633d84ca9bba5a2e3305bc3e8; no voters:
I20260928 14:56:49.367057 4729 leader_election.cc:288] T 00000000000000000000000000000000 P b634054633d84ca9bba5a2e3305bc3e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:56:49.367233 4734 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b634054633d84ca9bba5a2e3305bc3e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:56:49.368255 4734 raft_consensus.cc:686] T 00000000000000000000000000000000 P b634054633d84ca9bba5a2e3305bc3e8 [term 1 LEADER]: Becoming Leader. State: Replica: b634054633d84ca9bba5a2e3305bc3e8, State: Running, Role: LEADER
I20260928 14:56:49.368849 4734 consensus_queue.cc:237] T 00000000000000000000000000000000 P b634054633d84ca9bba5a2e3305bc3e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b634054633d84ca9bba5a2e3305bc3e8" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 42547 } }
I20260928 14:56:49.369540 4729 sys_catalog.cc:564] T 00000000000000000000000000000000 P b634054633d84ca9bba5a2e3305bc3e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:56:49.371706 4735 sys_catalog.cc:455] T 00000000000000000000000000000000 P b634054633d84ca9bba5a2e3305bc3e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b634054633d84ca9bba5a2e3305bc3e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b634054633d84ca9bba5a2e3305bc3e8" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 42547 } } }
I20260928 14:56:49.371809 4736 sys_catalog.cc:455] T 00000000000000000000000000000000 P b634054633d84ca9bba5a2e3305bc3e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b634054633d84ca9bba5a2e3305bc3e8. Latest consensus state: current_term: 1 leader_uuid: "b634054633d84ca9bba5a2e3305bc3e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b634054633d84ca9bba5a2e3305bc3e8" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 42547 } } }
I20260928 14:56:49.372256 4735 sys_catalog.cc:458] T 00000000000000000000000000000000 P b634054633d84ca9bba5a2e3305bc3e8 [sys.catalog]: This master's current role is: LEADER
I20260928 14:56:49.373124 4736 sys_catalog.cc:458] T 00000000000000000000000000000000 P b634054633d84ca9bba5a2e3305bc3e8 [sys.catalog]: This master's current role is: LEADER
I20260928 14:56:49.374279 4743 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:56:49.378957 4743 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:56:49.387259 4743 catalog_manager.cc:1347] Generated new cluster ID: 9a758fa4ac6f49ecaeea55490af9a2c3
I20260928 14:56:49.387382 4743 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:56:49.411412 4743 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:56:49.413457 4743 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:56:49.430538 4743 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b634054633d84ca9bba5a2e3305bc3e8: Generated new TSK 0
I20260928 14:56:49.431771 4743 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 14:56:49.488315 4759 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 14:56:49.488529 4759 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 14:56:49.489739 4732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:49.460105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52612 (local address 127.31.40.62:42547)
0928 14:56:49.460361 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:49.460371 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:49.460886 (+ 515us) server_negotiation.cc:407] Connection header received
0928 14:56:49.461194 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:49.461198 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:49.461267 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:49.461399 (+ 132us) server_negotiation.cc:226] Negotiated authn=JWT
0928 14:56:49.462085 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:49.462831 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:49.465060 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:49.465598 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:49.465714 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:49.466412 (+ 698us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 14:56:49.489001 (+ 22589us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46177/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 14:56:49.489409 (+ 408us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52612: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46177/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":76}
W20260928 14:56:49.489883 4732 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52612: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46177/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 14:56:49.491173 4758 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:56:49.460008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:42547 (local address 127.0.0.1:52612)
0928 14:56:49.460745 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:49.460769 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:49.460962 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:49.461464 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:49.461470 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:49.461483 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0928 14:56:49.461896 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:49.461908 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:49.462984 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:49.462991 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:49.464886 (+ 1895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:49.464903 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:49.465800 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:49.465807 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:49.465996 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:49.466113 (+ 117us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 14:56:49.489317 (+ 23204us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46177/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 14:56:49.489617 (+ 300us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.40.62:42547: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46177/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":353,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.31.40.62:42547: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46177/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 14:56:49.493592 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 4663
I20260928 14:56:49.494935 4724 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:56:49.595480 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 4663
2026-09-28T14:56:49Z chronyd exiting
I20260928 14:56:49.647536 31904 test_util.cc:175] -----------------------------------------------
I20260928 14:56:49.647701 31904 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607365362981-31904-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (804 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 14:56:49.652153 31904 mini_oidc.cc:152] Starting JWKS server
I20260928 14:56:49.654273 31904 webserver.cc:460] Webserver started at https://127.0.0.1:35779/ using document root <none> and password file <none>
I20260928 14:56:49.654542 31904 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 14:56:49.655272 31904 webserver.cc:460] Webserver started at http://127.0.0.1:39103/ using document root <none> and password file <none>
2026-09-28T14:56:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:56:49Z Disabled control of system clock
I20260928 14:56:49.684437 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607365362981-31904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607365362981-31904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607365362981-31904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:40199
--webserver_interface=127.31.40.62
--webserver_port=0
--builtin_ntp_servers=127.31.40.20:41189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.40.62:40199
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35779/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607365362981-31904-0/testchainca.cert with env {}
W20260928 14:56:50.021492 4766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:56:50.021812 4766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:56:50.021999 4766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:56:50.029405 4766 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:56:50.029527 4766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:56:50.029620 4766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:56:50.029728 4766 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:56:50.029816 4766 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 14:56:50.029937 4766 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 14:56:50.029992 4766 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35779/jwks.json
I20260928 14:56:50.040062 4766 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:41189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607365362981-31904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607365362981-31904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.40.62:40199
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:40199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35779/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.40.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607365362981-31904-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607365362981-31904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:56:50.041479 4766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:56:50.042882 4766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:56:50.055380 4774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:50.055476 4772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:50.055476 4771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:50.056715 4766 server_base.cc:1034] running on GCE node
I20260928 14:56:50.057758 4766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:56:50.059408 4766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:56:50.060670 4766 hybrid_clock.cc:648] HybridClock initialized: now 1790607410060621 us; error 75 us; skew 500 ppm
I20260928 14:56:50.061084 4766 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 14:56:50.063661 4766 webserver.cc:460] Webserver started at http://127.31.40.62:45631/ using document root <none> and password file <none>
I20260928 14:56:50.064443 4766 fs_manager.cc:359] Metadata directory not provided
I20260928 14:56:50.064579 4766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:56:50.064909 4766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:56:50.068035 4766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607365362981-31904-0/minicluster-data/master-0/data/instance:
uuid: "f7dc190eb1f4403e94c4a4681f1787b2"
format_stamp: "Formatted at 2026-09-28 14:56:50 on dist-test-slave-j968"
server_key: "ffdbe2d53cc9bf787037990de95e4980"
server_key_iv: "43a24e7dacd831ecf2ead67dfaa4add0"
server_key_version: "encryptionkey@0"
I20260928 14:56:50.068929 4766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607365362981-31904-0/minicluster-data/master-0/wal/instance:
uuid: "f7dc190eb1f4403e94c4a4681f1787b2"
format_stamp: "Formatted at 2026-09-28 14:56:50 on dist-test-slave-j968"
server_key: "ffdbe2d53cc9bf787037990de95e4980"
server_key_iv: "43a24e7dacd831ecf2ead67dfaa4add0"
server_key_version: "encryptionkey@0"
I20260928 14:56:50.074858 4766 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260928 14:56:50.079062 4780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:50.080830 4766 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:56:50.081072 4766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607365362981-31904-0/minicluster-data/master-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607365362981-31904-0/minicluster-data/master-0/wal
uuid: "f7dc190eb1f4403e94c4a4681f1787b2"
format_stamp: "Formatted at 2026-09-28 14:56:50 on dist-test-slave-j968"
server_key: "ffdbe2d53cc9bf787037990de95e4980"
server_key_iv: "43a24e7dacd831ecf2ead67dfaa4add0"
server_key_version: "encryptionkey@0"
I20260928 14:56:50.081336 4766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607365362981-31904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607365362981-31904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607365362981-31904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:56:50.116282 4766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:56:50.117264 4766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:56:50.117626 4766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:56:50.141893 4766 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.62:40199
I20260928 14:56:50.141927 4831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.62:40199 every 8 connection(s)
I20260928 14:56:50.143648 4766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
I20260928 14:56:50.145352 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 4766
I20260928 14:56:50.145546 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607365362981-31904-0/minicluster-data/master-0/wal/instance
I20260928 14:56:50.146391 31904 external_mini_cluster.cc:1428] Setting key d5f1c8ff16e395525a1db327c37463aa
I20260928 14:56:50.148955 4832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:56:50.160899 4835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:50.148625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36004 (local address 127.31.40.62:40199)
0928 14:56:50.149680 (+ 1055us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:50.149697 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:50.149739 (+ 42us) server_negotiation.cc:407] Connection header received
0928 14:56:50.151324 (+ 1585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:50.151344 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:50.151933 (+ 589us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:50.152308 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:50.152906 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:50.154298 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:50.156169 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:50.156909 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:50.157041 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:50.158288 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:50.158298 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:50.158312 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:56:50.158915 (+ 603us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:50.159001 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:56:50.159025 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:50.159123 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:50.159369 (+ 246us) server_negotiation.cc:299] Negotiation successful
0928 14:56:50.159791 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":414,"threads_started":1}
I20260928 14:56:50.162063 4832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7dc190eb1f4403e94c4a4681f1787b2: Bootstrap starting.
I20260928 14:56:50.164971 31904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:56:50.165632 4832 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7dc190eb1f4403e94c4a4681f1787b2: Neither blocks nor log segments found. Creating new log.
I20260928 14:56:50.166657 4832 log.cc:824] T 00000000000000000000000000000000 P f7dc190eb1f4403e94c4a4681f1787b2: Log is configured to *not* fsync() on all Append() calls
I20260928 14:56:50.169705 4832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7dc190eb1f4403e94c4a4681f1787b2: No bootstrap required, opened a new log
I20260928 14:56:50.176910 4832 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7dc190eb1f4403e94c4a4681f1787b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7dc190eb1f4403e94c4a4681f1787b2" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 40199 } }
I20260928 14:56:50.177300 4832 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7dc190eb1f4403e94c4a4681f1787b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:56:50.177407 4832 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7dc190eb1f4403e94c4a4681f1787b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7dc190eb1f4403e94c4a4681f1787b2, State: Initialized, Role: FOLLOWER
I20260928 14:56:50.177994 4832 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7dc190eb1f4403e94c4a4681f1787b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7dc190eb1f4403e94c4a4681f1787b2" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 40199 } }
I20260928 14:56:50.178198 4832 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7dc190eb1f4403e94c4a4681f1787b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:56:50.178371 4832 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7dc190eb1f4403e94c4a4681f1787b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:56:50.178562 4832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7dc190eb1f4403e94c4a4681f1787b2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:56:50.180830 4832 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7dc190eb1f4403e94c4a4681f1787b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7dc190eb1f4403e94c4a4681f1787b2" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 40199 } }
I20260928 14:56:50.181363 4832 leader_election.cc:302] T 00000000000000000000000000000000 P f7dc190eb1f4403e94c4a4681f1787b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7dc190eb1f4403e94c4a4681f1787b2; no voters:
I20260928 14:56:50.181953 4832 leader_election.cc:288] T 00000000000000000000000000000000 P f7dc190eb1f4403e94c4a4681f1787b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:56:50.182139 4837 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7dc190eb1f4403e94c4a4681f1787b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:56:50.183355 4837 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7dc190eb1f4403e94c4a4681f1787b2 [term 1 LEADER]: Becoming Leader. State: Replica: f7dc190eb1f4403e94c4a4681f1787b2, State: Running, Role: LEADER
I20260928 14:56:50.183852 4837 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7dc190eb1f4403e94c4a4681f1787b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7dc190eb1f4403e94c4a4681f1787b2" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 40199 } }
I20260928 14:56:50.184271 4832 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7dc190eb1f4403e94c4a4681f1787b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:56:50.186797 4839 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7dc190eb1f4403e94c4a4681f1787b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7dc190eb1f4403e94c4a4681f1787b2. Latest consensus state: current_term: 1 leader_uuid: "f7dc190eb1f4403e94c4a4681f1787b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7dc190eb1f4403e94c4a4681f1787b2" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 40199 } } }
I20260928 14:56:50.186901 4838 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7dc190eb1f4403e94c4a4681f1787b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7dc190eb1f4403e94c4a4681f1787b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7dc190eb1f4403e94c4a4681f1787b2" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 40199 } } }
I20260928 14:56:50.187292 4838 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7dc190eb1f4403e94c4a4681f1787b2 [sys.catalog]: This master's current role is: LEADER
I20260928 14:56:50.187292 4839 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7dc190eb1f4403e94c4a4681f1787b2 [sys.catalog]: This master's current role is: LEADER
I20260928 14:56:50.188242 4844 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:56:50.193835 4844 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:56:50.203567 4844 catalog_manager.cc:1347] Generated new cluster ID: 78d645af53c54a12ae9ef31a442ea7b1
I20260928 14:56:50.203694 4844 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:56:50.222847 4844 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:56:50.224352 4844 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:56:50.232184 4844 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7dc190eb1f4403e94c4a4681f1787b2: Generated new TSK 0
I20260928 14:56:50.233065 4844 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 14:56:50.270756 4862 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 14:56:50.270898 4862 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 14:56:50.272293 4835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:50.251536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36018 (local address 127.31.40.62:40199)
0928 14:56:50.251727 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:50.251734 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:50.252145 (+ 411us) server_negotiation.cc:407] Connection header received
0928 14:56:50.252360 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:50.252367 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:50.252468 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:50.252661 (+ 193us) server_negotiation.cc:226] Negotiated authn=JWT
0928 14:56:50.253188 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:50.254288 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:50.255825 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:50.256608 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:50.256737 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:50.257281 (+ 544us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 14:56:50.271498 (+ 14217us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35779/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 14:56:50.271946 (+ 448us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36018: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35779/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":73}
W20260928 14:56:50.272425 4835 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36018: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35779/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260928 14:56:50.272565 4861 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:56:50.251384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40199 (local address 127.0.0.1:36018)
0928 14:56:50.252042 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:50.252058 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:50.252183 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:50.252679 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:50.252682 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:50.252692 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0928 14:56:50.252962 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:50.253015 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:50.254446 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:50.254450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:50.255666 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:50.255680 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:50.256770 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:50.256774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:50.256917 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:50.256955 (+ 38us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 14:56:50.271924 (+ 14969us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35779/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 14:56:50.272167 (+ 243us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.40.62:40199: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35779/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":353,"threads_started":1}
I20260928 14:56:50.274569 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 4766
I20260928 14:56:50.275787 4827 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:56:50.372757 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 4766
2026-09-28T14:56:50Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (777 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 14:56:50.429781 31904 mini_oidc.cc:152] Starting JWKS server
I20260928 14:56:50.431777 31904 webserver.cc:460] Webserver started at https://127.0.0.1:41333/ using document root <none> and password file <none>
I20260928 14:56:50.432420 31904 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 14:56:50.433415 31904 webserver.cc:460] Webserver started at http://127.0.0.1:36701/ using document root <none> and password file <none>
2026-09-28T14:56:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:56:50Z Disabled control of system clock
I20260928 14:56:50.463353 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607365362981-31904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607365362981-31904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607365362981-31904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:36081
--webserver_interface=127.31.40.62
--webserver_port=0
--builtin_ntp_servers=127.31.40.20:33471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.40.62:36081
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41333/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 14:56:50.806361 4869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:56:50.806675 4869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:56:50.806826 4869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:56:50.814414 4869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:56:50.814535 4869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:56:50.814604 4869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:56:50.814706 4869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:56:50.814767 4869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 14:56:50.814888 4869 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 14:56:50.814962 4869 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41333/jwks.json
I20260928 14:56:50.824950 4869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:33471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607365362981-31904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607365362981-31904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.40.62:36081
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:36081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41333/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607365362981-31904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:56:50.826381 4869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:56:50.828383 4869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:56:50.841406 4875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:50.841351 4877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:50.841437 4874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:50.842046 4869 server_base.cc:1034] running on GCE node
I20260928 14:56:50.843014 4869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:56:50.844643 4869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:56:50.845866 4869 hybrid_clock.cc:648] HybridClock initialized: now 1790607410845815 us; error 41 us; skew 500 ppm
I20260928 14:56:50.846278 4869 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 14:56:50.848774 4869 webserver.cc:460] Webserver started at http://127.31.40.62:39617/ using document root <none> and password file <none>
I20260928 14:56:50.849467 4869 fs_manager.cc:359] Metadata directory not provided
I20260928 14:56:50.849556 4869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:56:50.849838 4869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:56:50.852919 4869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607365362981-31904-0/minicluster-data/master-0/data/instance:
uuid: "18aeacc4730141b8903b702aec945b0c"
format_stamp: "Formatted at 2026-09-28 14:56:50 on dist-test-slave-j968"
server_key: "b737f2593993d7282e938ac060e0513f"
server_key_iv: "b646a6b990420bfd4bfcac97393f7cbd"
server_key_version: "encryptionkey@0"
I20260928 14:56:50.853710 4869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607365362981-31904-0/minicluster-data/master-0/wal/instance:
uuid: "18aeacc4730141b8903b702aec945b0c"
format_stamp: "Formatted at 2026-09-28 14:56:50 on dist-test-slave-j968"
server_key: "b737f2593993d7282e938ac060e0513f"
server_key_iv: "b646a6b990420bfd4bfcac97393f7cbd"
server_key_version: "encryptionkey@0"
I20260928 14:56:50.858954 4869 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 14:56:50.862633 4883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:50.864327 4869 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:56:50.864504 4869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607365362981-31904-0/minicluster-data/master-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607365362981-31904-0/minicluster-data/master-0/wal
uuid: "18aeacc4730141b8903b702aec945b0c"
format_stamp: "Formatted at 2026-09-28 14:56:50 on dist-test-slave-j968"
server_key: "b737f2593993d7282e938ac060e0513f"
server_key_iv: "b646a6b990420bfd4bfcac97393f7cbd"
server_key_version: "encryptionkey@0"
I20260928 14:56:50.864702 4869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607365362981-31904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607365362981-31904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607365362981-31904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:56:50.880523 4869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:56:50.881320 4869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:56:50.881575 4869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:56:50.899897 4869 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.62:36081
I20260928 14:56:50.899906 4934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.62:36081 every 8 connection(s)
I20260928 14:56:50.901365 4869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
I20260928 14:56:50.904464 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 4869
I20260928 14:56:50.904666 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607365362981-31904-0/minicluster-data/master-0/wal/instance
I20260928 14:56:50.905265 4935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:56:50.905339 31904 external_mini_cluster.cc:1428] Setting key 9d1dd87313b9fd0204b9a0ea4aca7b15
I20260928 14:56:50.918089 4935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18aeacc4730141b8903b702aec945b0c: Bootstrap starting.
I20260928 14:56:50.919121 4938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:50.907312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56422 (local address 127.31.40.62:36081)
0928 14:56:50.908386 (+ 1074us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:50.908399 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:50.908451 (+ 52us) server_negotiation.cc:407] Connection header received
0928 14:56:50.910108 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:50.910130 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:50.910622 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:50.911043 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:50.911624 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:50.912983 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:50.914289 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:50.914983 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:50.915196 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:50.916391 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:50.916399 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:50.916412 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:56:50.916936 (+ 524us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:50.917010 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:56:50.917031 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:50.917160 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:50.917378 (+ 218us) server_negotiation.cc:299] Negotiation successful
0928 14:56:50.917745 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":350,"threads_started":1}
I20260928 14:56:50.922034 4935 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18aeacc4730141b8903b702aec945b0c: Neither blocks nor log segments found. Creating new log.
I20260928 14:56:50.923322 4935 log.cc:824] T 00000000000000000000000000000000 P 18aeacc4730141b8903b702aec945b0c: Log is configured to *not* fsync() on all Append() calls
I20260928 14:56:50.923585 31904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:56:50.926756 4935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18aeacc4730141b8903b702aec945b0c: No bootstrap required, opened a new log
I20260928 14:56:50.934553 4935 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18aeacc4730141b8903b702aec945b0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18aeacc4730141b8903b702aec945b0c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 36081 } }
I20260928 14:56:50.935048 4935 raft_consensus.cc:374] T 00000000000000000000000000000000 P 18aeacc4730141b8903b702aec945b0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:56:50.935163 4935 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18aeacc4730141b8903b702aec945b0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18aeacc4730141b8903b702aec945b0c, State: Initialized, Role: FOLLOWER
I20260928 14:56:50.935772 4935 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18aeacc4730141b8903b702aec945b0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18aeacc4730141b8903b702aec945b0c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 36081 } }
I20260928 14:56:50.936065 4935 raft_consensus.cc:388] T 00000000000000000000000000000000 P 18aeacc4730141b8903b702aec945b0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:56:50.936206 4935 raft_consensus.cc:482] T 00000000000000000000000000000000 P 18aeacc4730141b8903b702aec945b0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:56:50.936388 4935 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18aeacc4730141b8903b702aec945b0c [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:56:50.938568 4935 raft_consensus.cc:504] T 00000000000000000000000000000000 P 18aeacc4730141b8903b702aec945b0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18aeacc4730141b8903b702aec945b0c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 36081 } }
I20260928 14:56:50.939086 4935 leader_election.cc:302] T 00000000000000000000000000000000 P 18aeacc4730141b8903b702aec945b0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 18aeacc4730141b8903b702aec945b0c; no voters:
I20260928 14:56:50.939803 4935 leader_election.cc:288] T 00000000000000000000000000000000 P 18aeacc4730141b8903b702aec945b0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:56:50.939981 4940 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 18aeacc4730141b8903b702aec945b0c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:56:50.940681 4940 raft_consensus.cc:686] T 00000000000000000000000000000000 P 18aeacc4730141b8903b702aec945b0c [term 1 LEADER]: Becoming Leader. State: Replica: 18aeacc4730141b8903b702aec945b0c, State: Running, Role: LEADER
I20260928 14:56:50.941236 4940 consensus_queue.cc:237] T 00000000000000000000000000000000 P 18aeacc4730141b8903b702aec945b0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18aeacc4730141b8903b702aec945b0c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 36081 } }
I20260928 14:56:50.942186 4935 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18aeacc4730141b8903b702aec945b0c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:56:50.944325 4941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18aeacc4730141b8903b702aec945b0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "18aeacc4730141b8903b702aec945b0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18aeacc4730141b8903b702aec945b0c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 36081 } } }
I20260928 14:56:50.944419 4942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18aeacc4730141b8903b702aec945b0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18aeacc4730141b8903b702aec945b0c. Latest consensus state: current_term: 1 leader_uuid: "18aeacc4730141b8903b702aec945b0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18aeacc4730141b8903b702aec945b0c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 36081 } } }
I20260928 14:56:50.944778 4942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18aeacc4730141b8903b702aec945b0c [sys.catalog]: This master's current role is: LEADER
I20260928 14:56:50.944784 4941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18aeacc4730141b8903b702aec945b0c [sys.catalog]: This master's current role is: LEADER
I20260928 14:56:50.947510 4947 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:56:50.952742 4947 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:56:50.961879 4947 catalog_manager.cc:1347] Generated new cluster ID: 58cd59ddf74f4d95a654a35da8eb34c3
I20260928 14:56:50.962005 4947 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:56:50.976781 4947 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:56:50.978369 4947 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:56:50.994549 4947 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 18aeacc4730141b8903b702aec945b0c: Generated new TSK 0
I20260928 14:56:50.995544 4947 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:56:51.026607 4938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:51.018002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56432 (local address 127.31.40.62:36081)
0928 14:56:51.018193 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:51.018200 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:51.018774 (+ 574us) server_negotiation.cc:407] Connection header received
0928 14:56:51.019086 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:51.019090 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:51.019174 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:51.019324 (+ 150us) server_negotiation.cc:226] Negotiated authn=JWT
0928 14:56:51.019988 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:51.020721 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:51.022735 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:51.023159 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:51.023252 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:51.023828 (+ 576us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 14:56:51.025917 (+ 2089us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41333/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 14:56:51.026278 (+ 361us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56432: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41333/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":50}
W20260928 14:56:51.026744 4938 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56432: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41333/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 14:56:51.026834 4964 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:56:51.017882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:36081 (local address 127.0.0.1:56432)
0928 14:56:51.018653 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:51.018673 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:51.018837 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:51.019378 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:51.019384 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:51.019398 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0928 14:56:51.019709 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:51.019721 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:51.020870 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:51.020876 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:51.022591 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:51.022606 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:51.023343 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:51.023350 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:51.023530 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:51.023577 (+ 47us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 14:56:51.026128 (+ 2551us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41333/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 14:56:51.026393 (+ 265us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.40.62:36081: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41333/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":447,"threads_started":1}
I20260928 14:56:51.027923 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 4869
I20260928 14:56:51.029132 4930 generic_service.cc:196] Checking for leaks (request via RPC)
E20260928 14:56:51.039163 4965 webserver.cc:533] Webserver: sslize failed:
E20260928 14:56:51.039290 4965 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 14:56:51.122731 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 4869
2026-09-28T14:56:51Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (749 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T14:56:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:56:51Z Disabled control of system clock
I20260928 14:56:51.206823 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607365362981-31904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607365362981-31904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607365362981-31904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:44067
--webserver_interface=127.31.40.62
--webserver_port=0
--builtin_ntp_servers=127.31.40.20:38369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.40.62:44067
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607365362981-31904-0/testchainca.cert with env {}
W20260928 14:56:51.539963 4970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:56:51.540269 4970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:56:51.540433 4970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:56:51.549044 4970 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:56:51.549158 4970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:56:51.549244 4970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:56:51.549342 4970 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:56:51.549422 4970 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 14:56:51.549510 4970 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 14:56:51.549582 4970 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 14:56:51.560194 4970 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:38369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607365362981-31904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607365362981-31904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.40.62:44067
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:44067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.40.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607365362981-31904-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607365362981-31904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:56:51.561602 4970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:56:51.562980 4970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:56:51.576413 4975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:51.576388 4978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:51.576351 4976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:51.577389 4970 server_base.cc:1034] running on GCE node
I20260928 14:56:51.578284 4970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:56:51.579802 4970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:56:51.581096 4970 hybrid_clock.cc:648] HybridClock initialized: now 1790607411581026 us; error 65 us; skew 500 ppm
I20260928 14:56:51.581501 4970 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 14:56:51.584043 4970 webserver.cc:460] Webserver started at http://127.31.40.62:39341/ using document root <none> and password file <none>
I20260928 14:56:51.584766 4970 fs_manager.cc:359] Metadata directory not provided
I20260928 14:56:51.584888 4970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:56:51.585210 4970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:56:51.588155 4970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607365362981-31904-0/minicluster-data/master-0/data/instance:
uuid: "99a941f4ea1c48bc920e6bd7a31654a7"
format_stamp: "Formatted at 2026-09-28 14:56:51 on dist-test-slave-j968"
server_key: "4ee864d218b78b396c30dafbe6b182a4"
server_key_iv: "7c3b81e13d4dea32cb49c53dbf66f3b1"
server_key_version: "encryptionkey@0"
I20260928 14:56:51.588950 4970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607365362981-31904-0/minicluster-data/master-0/wal/instance:
uuid: "99a941f4ea1c48bc920e6bd7a31654a7"
format_stamp: "Formatted at 2026-09-28 14:56:51 on dist-test-slave-j968"
server_key: "4ee864d218b78b396c30dafbe6b182a4"
server_key_iv: "7c3b81e13d4dea32cb49c53dbf66f3b1"
server_key_version: "encryptionkey@0"
I20260928 14:56:51.594372 4970 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 14:56:51.598012 4984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:51.599587 4970 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:56:51.599936 4970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607365362981-31904-0/minicluster-data/master-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607365362981-31904-0/minicluster-data/master-0/wal
uuid: "99a941f4ea1c48bc920e6bd7a31654a7"
format_stamp: "Formatted at 2026-09-28 14:56:51 on dist-test-slave-j968"
server_key: "4ee864d218b78b396c30dafbe6b182a4"
server_key_iv: "7c3b81e13d4dea32cb49c53dbf66f3b1"
server_key_version: "encryptionkey@0"
I20260928 14:56:51.600200 4970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607365362981-31904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607365362981-31904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607365362981-31904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:56:51.615594 4970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:56:51.616560 4970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:56:51.616863 4970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:56:51.636000 4970 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.62:44067
I20260928 14:56:51.635970 5035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.62:44067 every 8 connection(s)
I20260928 14:56:51.637391 4970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
I20260928 14:56:51.641228 5036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:56:51.647991 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 4970
I20260928 14:56:51.648228 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607365362981-31904-0/minicluster-data/master-0/wal/instance
I20260928 14:56:51.648914 31904 external_mini_cluster.cc:1428] Setting key 64c24ef8329da113461af0d1cc9ba88e
I20260928 14:56:51.652014 5036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99a941f4ea1c48bc920e6bd7a31654a7: Bootstrap starting.
I20260928 14:56:51.655326 5036 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99a941f4ea1c48bc920e6bd7a31654a7: Neither blocks nor log segments found. Creating new log.
I20260928 14:56:51.656487 5036 log.cc:824] T 00000000000000000000000000000000 P 99a941f4ea1c48bc920e6bd7a31654a7: Log is configured to *not* fsync() on all Append() calls
I20260928 14:56:51.659927 5036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99a941f4ea1c48bc920e6bd7a31654a7: No bootstrap required, opened a new log
I20260928 14:56:51.664345 5039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:51.651033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33104 (local address 127.31.40.62:44067)
0928 14:56:51.652251 (+ 1218us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:51.652263 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:51.652310 (+ 47us) server_negotiation.cc:407] Connection header received
0928 14:56:51.654147 (+ 1837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:51.654174 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:51.654670 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:51.655030 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:51.655717 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:51.657560 (+ 1843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:51.659397 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:51.660216 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:51.660357 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:51.661576 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:51.661584 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:51.661598 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:56:51.662430 (+ 832us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:51.662560 (+ 130us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:56:51.662593 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:51.662713 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:51.663062 (+ 349us) server_negotiation.cc:299] Negotiation successful
0928 14:56:51.663314 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":389,"threads_started":1}
I20260928 14:56:51.669095 31904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:56:51.670225 5036 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99a941f4ea1c48bc920e6bd7a31654a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99a941f4ea1c48bc920e6bd7a31654a7" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 44067 } }
I20260928 14:56:51.670614 5036 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99a941f4ea1c48bc920e6bd7a31654a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:56:51.670730 5036 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99a941f4ea1c48bc920e6bd7a31654a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a941f4ea1c48bc920e6bd7a31654a7, State: Initialized, Role: FOLLOWER
I20260928 14:56:51.671384 5036 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99a941f4ea1c48bc920e6bd7a31654a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99a941f4ea1c48bc920e6bd7a31654a7" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 44067 } }
I20260928 14:56:51.671671 5036 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99a941f4ea1c48bc920e6bd7a31654a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:56:51.671824 5036 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99a941f4ea1c48bc920e6bd7a31654a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:56:51.672091 5036 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99a941f4ea1c48bc920e6bd7a31654a7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:56:51.674629 5036 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99a941f4ea1c48bc920e6bd7a31654a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99a941f4ea1c48bc920e6bd7a31654a7" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 44067 } }
I20260928 14:56:51.675354 5036 leader_election.cc:302] T 00000000000000000000000000000000 P 99a941f4ea1c48bc920e6bd7a31654a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99a941f4ea1c48bc920e6bd7a31654a7; no voters:
I20260928 14:56:51.676137 5036 leader_election.cc:288] T 00000000000000000000000000000000 P 99a941f4ea1c48bc920e6bd7a31654a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:56:51.676246 5041 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99a941f4ea1c48bc920e6bd7a31654a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:56:51.677731 5041 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99a941f4ea1c48bc920e6bd7a31654a7 [term 1 LEADER]: Becoming Leader. State: Replica: 99a941f4ea1c48bc920e6bd7a31654a7, State: Running, Role: LEADER
I20260928 14:56:51.678299 5041 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99a941f4ea1c48bc920e6bd7a31654a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99a941f4ea1c48bc920e6bd7a31654a7" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 44067 } }
I20260928 14:56:51.678442 5036 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99a941f4ea1c48bc920e6bd7a31654a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:56:51.681165 5043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99a941f4ea1c48bc920e6bd7a31654a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99a941f4ea1c48bc920e6bd7a31654a7. Latest consensus state: current_term: 1 leader_uuid: "99a941f4ea1c48bc920e6bd7a31654a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99a941f4ea1c48bc920e6bd7a31654a7" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 44067 } } }
I20260928 14:56:51.681545 5043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99a941f4ea1c48bc920e6bd7a31654a7 [sys.catalog]: This master's current role is: LEADER
I20260928 14:56:51.682122 5042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99a941f4ea1c48bc920e6bd7a31654a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99a941f4ea1c48bc920e6bd7a31654a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99a941f4ea1c48bc920e6bd7a31654a7" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 44067 } } }
I20260928 14:56:51.682451 5042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99a941f4ea1c48bc920e6bd7a31654a7 [sys.catalog]: This master's current role is: LEADER
I20260928 14:56:51.684697 5049 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:56:51.691332 5049 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:56:51.701176 5049 catalog_manager.cc:1347] Generated new cluster ID: 38e087ff95cb49f6b36bac43fd1ddba5
I20260928 14:56:51.701299 5049 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:56:51.723011 5049 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:56:51.724602 5049 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:56:51.738840 5049 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99a941f4ea1c48bc920e6bd7a31654a7: Generated new TSK 0
I20260928 14:56:51.740317 5049 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:56:51.788829 5039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:51.760623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33114 (local address 127.31.40.62:44067)
0928 14:56:51.760842 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:51.760850 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:51.761305 (+ 455us) server_negotiation.cc:407] Connection header received
0928 14:56:51.761531 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:51.761538 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:51.761635 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:51.761814 (+ 179us) server_negotiation.cc:226] Negotiated authn=JWT
0928 14:56:51.762336 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:51.763495 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:51.764983 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:51.765695 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:51.765826 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:51.766363 (+ 537us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 14:56:51.788016 (+ 21653us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 14:56:51.788479 (+ 463us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33114: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":71}
W20260928 14:56:51.788972 5039 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33114: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 14:56:51.789069 5066 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:56:51.760468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:44067 (local address 127.0.0.1:33114)
0928 14:56:51.761194 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:51.761212 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:51.761344 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:51.761847 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:51.761850 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:51.761874 (+ 24us) client_negotiation.cc:189] Negotiated authn=JWT
0928 14:56:51.762174 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:51.762183 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:51.763659 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:51.763663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:51.764840 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:51.764852 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:51.765873 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:51.765877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:51.766044 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:51.766083 (+ 39us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 14:56:51.788384 (+ 22301us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 14:56:51.788711 (+ 327us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.40.62:44067: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":410,"threads_started":1}
I20260928 14:56:51.790427 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 4970
I20260928 14:56:51.791939 5031 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:56:51.887362 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 4970
2026-09-28T14:56:51Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (746 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (424 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (422 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:56:52 dist-test-slave-j968 krb5kdc[5081](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:56:52 dist-test-slave-j968 krb5kdc[5081](info): set up 2 sockets
Sep 28 14:56:52 dist-test-slave-j968 krb5kdc[5081](info): commencing operation
krb5kdc: starting...
W20260928 14:56:53.834009 31904 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.002s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Sep 28 14:56:53 dist-test-slave-j968 krb5kdc[5081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:56:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:56:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab.
Entry for principal kudu/127.31.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab.
I20260928 14:56:54.057746 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:33541
--webserver_interface=127.31.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:34117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.40.62:33541
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:56:54.430320 5094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:56:54.430681 5094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:56:54.430786 5094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:56:54.439360 5094 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:56:54.439494 5094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:56:54.439543 5094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:56:54.439594 5094 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:56:54.439639 5094 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:56:54.451829 5094 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:34117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.40.62:33541
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:33541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:56:54.453382 5094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:56:54.454833 5094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:56:54.468271 5099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:54.468405 5102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:54.468536 5100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:54.469998 5094 server_base.cc:1034] running on GCE node
I20260928 14:56:54.471014 5094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:56:54.472685 5094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:56:54.473970 5094 hybrid_clock.cc:648] HybridClock initialized: now 1790607414473898 us; error 58 us; skew 500 ppm
I20260928 14:56:54.474383 5094 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:56:54 dist-test-slave-j968 krb5kdc[5081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607414, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:56:54.477850 5094 init.cc:377] Logged in from keytab as kudu/127.31.40.62@KRBTEST.COM (short username kudu)
I20260928 14:56:54.479532 5094 webserver.cc:460] Webserver started at http://127.31.40.62:41665/ using document root <none> and password file <none>
I20260928 14:56:54.480358 5094 fs_manager.cc:359] Metadata directory not provided
I20260928 14:56:54.480517 5094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:56:54.480841 5094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:56:54.484018 5094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/master-0/data/instance:
uuid: "165efef742c14adda9997e28a3438cd3"
format_stamp: "Formatted at 2026-09-28 14:56:54 on dist-test-slave-j968"
server_key: "db722f50124b0a14476c9edef604b822"
server_key_iv: "f1ab9d5dd60af32b835e4925d8a5e6dc"
server_key_version: "encryptionkey@0"
I20260928 14:56:54.484843 5094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/master-0/wal/instance:
uuid: "165efef742c14adda9997e28a3438cd3"
format_stamp: "Formatted at 2026-09-28 14:56:54 on dist-test-slave-j968"
server_key: "db722f50124b0a14476c9edef604b822"
server_key_iv: "f1ab9d5dd60af32b835e4925d8a5e6dc"
server_key_version: "encryptionkey@0"
I20260928 14:56:54.490558 5094 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 14:56:54.494671 5109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:54.496297 5094 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 14:56:54.496628 5094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/master-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/master-0/wal
uuid: "165efef742c14adda9997e28a3438cd3"
format_stamp: "Formatted at 2026-09-28 14:56:54 on dist-test-slave-j968"
server_key: "db722f50124b0a14476c9edef604b822"
server_key_iv: "f1ab9d5dd60af32b835e4925d8a5e6dc"
server_key_version: "encryptionkey@0"
I20260928 14:56:54.496873 5094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:56:54.513062 5094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:56:54.518371 5094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:56:54.518728 5094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:56:54.540478 5094 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.62:33541
I20260928 14:56:54.540485 5160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.62:33541 every 8 connection(s)
I20260928 14:56:54.542030 5094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
I20260928 14:56:54.546459 5161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:56:54.550839 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 5094
I20260928 14:56:54.551018 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/master-0/wal/instance
I20260928 14:56:54.551694 31904 external_mini_cluster.cc:1428] Setting key f158057a3861203e6d46b4f4dc2e9208
I20260928 14:56:54.562726 5161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 165efef742c14adda9997e28a3438cd3: Bootstrap starting.
I20260928 14:56:54.567180 5161 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 165efef742c14adda9997e28a3438cd3: Neither blocks nor log segments found. Creating new log.
I20260928 14:56:54.568799 5161 log.cc:824] T 00000000000000000000000000000000 P 165efef742c14adda9997e28a3438cd3: Log is configured to *not* fsync() on all Append() calls
Sep 28 14:56:54 dist-test-slave-j968 krb5kdc[5081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:56:54.573156 5161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 165efef742c14adda9997e28a3438cd3: No bootstrap required, opened a new log
I20260928 14:56:54.581550 5164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:54.554012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39026 (local address 127.31.40.62:33541)
0928 14:56:54.555156 (+ 1144us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:54.555179 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:54.555241 (+ 62us) server_negotiation.cc:407] Connection header received
0928 14:56:54.557624 (+ 2383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:54.557661 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:54.558306 (+ 645us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:54.558857 (+ 551us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:54.561590 (+ 2733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:54.563161 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:54.565130 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:54.565890 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:54.566052 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:54.572039 (+ 5987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:54.572049 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:54.572067 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:54.572147 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:54.576048 (+ 3901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:54.577271 (+ 1223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:54.577281 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:54.577288 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:54.577521 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:54.578205 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:54.578209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:54.578212 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:54.578869 (+ 657us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:54.579143 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:54.579684 (+ 541us) server_negotiation.cc:299] Negotiation successful
0928 14:56:54.580366 (+ 682us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":739,"thread_start_us":507,"threads_started":1}
I20260928 14:56:54.583762 5161 raft_consensus.cc:348] T 00000000000000000000000000000000 P 165efef742c14adda9997e28a3438cd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "165efef742c14adda9997e28a3438cd3" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 33541 } }
I20260928 14:56:54.584297 5161 raft_consensus.cc:374] T 00000000000000000000000000000000 P 165efef742c14adda9997e28a3438cd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:56:54.584451 5161 raft_consensus.cc:729] T 00000000000000000000000000000000 P 165efef742c14adda9997e28a3438cd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 165efef742c14adda9997e28a3438cd3, State: Initialized, Role: FOLLOWER
I20260928 14:56:54.585240 5161 consensus_queue.cc:260] T 00000000000000000000000000000000 P 165efef742c14adda9997e28a3438cd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "165efef742c14adda9997e28a3438cd3" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 33541 } }
I20260928 14:56:54.585613 5161 raft_consensus.cc:388] T 00000000000000000000000000000000 P 165efef742c14adda9997e28a3438cd3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:56:54.585793 5161 raft_consensus.cc:482] T 00000000000000000000000000000000 P 165efef742c14adda9997e28a3438cd3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:56:54.586005 5161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 165efef742c14adda9997e28a3438cd3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:56:54.588963 5161 raft_consensus.cc:504] T 00000000000000000000000000000000 P 165efef742c14adda9997e28a3438cd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "165efef742c14adda9997e28a3438cd3" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 33541 } }
I20260928 14:56:54.589692 5161 leader_election.cc:302] T 00000000000000000000000000000000 P 165efef742c14adda9997e28a3438cd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 165efef742c14adda9997e28a3438cd3; no voters:
I20260928 14:56:54.590420 5161 leader_election.cc:288] T 00000000000000000000000000000000 P 165efef742c14adda9997e28a3438cd3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:56:54.590525 5166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 165efef742c14adda9997e28a3438cd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:56:54.591346 5166 raft_consensus.cc:686] T 00000000000000000000000000000000 P 165efef742c14adda9997e28a3438cd3 [term 1 LEADER]: Becoming Leader. State: Replica: 165efef742c14adda9997e28a3438cd3, State: Running, Role: LEADER
I20260928 14:56:54.592119 5166 consensus_queue.cc:237] T 00000000000000000000000000000000 P 165efef742c14adda9997e28a3438cd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "165efef742c14adda9997e28a3438cd3" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 33541 } }
I20260928 14:56:54.592998 5161 sys_catalog.cc:564] T 00000000000000000000000000000000 P 165efef742c14adda9997e28a3438cd3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:56:54.595088 5167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 165efef742c14adda9997e28a3438cd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "165efef742c14adda9997e28a3438cd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "165efef742c14adda9997e28a3438cd3" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 33541 } } }
I20260928 14:56:54.595578 5167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 165efef742c14adda9997e28a3438cd3 [sys.catalog]: This master's current role is: LEADER
I20260928 14:56:54.596100 5168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 165efef742c14adda9997e28a3438cd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 165efef742c14adda9997e28a3438cd3. Latest consensus state: current_term: 1 leader_uuid: "165efef742c14adda9997e28a3438cd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "165efef742c14adda9997e28a3438cd3" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 33541 } } }
I20260928 14:56:54.596474 5168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 165efef742c14adda9997e28a3438cd3 [sys.catalog]: This master's current role is: LEADER
I20260928 14:56:54.600586 5175 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:56:54.608301 5175 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:56:54.620942 5175 catalog_manager.cc:1347] Generated new cluster ID: 69b103b4ac7548eaafdd2e38049d74f3
I20260928 14:56:54.621088 5175 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:56:54.631366 5175 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:56:54.633047 5175 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:56:54.648494 5175 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 165efef742c14adda9997e28a3438cd3: Generated new TSK 0
I20260928 14:56:54.649576 5175 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.40.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab.
Entry for principal kudu/127.31.40.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab.
I20260928 14:56:54.708417 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.1:0
--local_ip_for_outbound_sockets=127.31.40.1
--webserver_interface=127.31.40.1
--webserver_port=0
--tserver_master_addrs=127.31.40.62:33541
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab
--principal=kudu/127.31.40.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:34117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:56:55.061920 5187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:56:55.062273 5187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:56:55.062438 5187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:56:55.069813 5187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:56:55.069993 5187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:56:55.070170 5187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.1
I20260928 14:56:55.080621 5187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:34117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab
--principal=kudu/127.31.40.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.40.1
--webserver_port=0
--tserver_master_addrs=127.31.40.62:33541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.1
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:56:55.082085 5187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:56:55.083457 5187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:56:55.097532 5195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:55.097452 5193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:55.098099 5192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:55.098627 5187 server_base.cc:1034] running on GCE node
I20260928 14:56:55.099438 5187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:56:55.100718 5187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:56:55.101999 5187 hybrid_clock.cc:648] HybridClock initialized: now 1790607415101889 us; error 94 us; skew 500 ppm
I20260928 14:56:55.102383 5187 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:56:55 dist-test-slave-j968 krb5kdc[5081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607415, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:56:55.105930 5187 init.cc:377] Logged in from keytab as kudu/127.31.40.1@KRBTEST.COM (short username kudu)
I20260928 14:56:55.107482 5187 webserver.cc:460] Webserver started at http://127.31.40.1:39711/ using document root <none> and password file <none>
I20260928 14:56:55.108259 5187 fs_manager.cc:359] Metadata directory not provided
I20260928 14:56:55.108407 5187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:56:55.108724 5187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:56:55.111672 5187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/ts-0/data/instance:
uuid: "91541ec1a80e4bfcbe441eed71dc00c0"
format_stamp: "Formatted at 2026-09-28 14:56:55 on dist-test-slave-j968"
server_key: "52a17205ed94236985be9162a25e6965"
server_key_iv: "2e17221480f1f59b7be356959415b63e"
server_key_version: "encryptionkey@0"
I20260928 14:56:55.112526 5187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/ts-0/wal/instance:
uuid: "91541ec1a80e4bfcbe441eed71dc00c0"
format_stamp: "Formatted at 2026-09-28 14:56:55 on dist-test-slave-j968"
server_key: "52a17205ed94236985be9162a25e6965"
server_key_iv: "2e17221480f1f59b7be356959415b63e"
server_key_version: "encryptionkey@0"
I20260928 14:56:55.118162 5187 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 14:56:55.122308 5202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:55.124084 5187 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 14:56:55.124286 5187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/ts-0/wal
uuid: "91541ec1a80e4bfcbe441eed71dc00c0"
format_stamp: "Formatted at 2026-09-28 14:56:55 on dist-test-slave-j968"
server_key: "52a17205ed94236985be9162a25e6965"
server_key_iv: "2e17221480f1f59b7be356959415b63e"
server_key_version: "encryptionkey@0"
I20260928 14:56:55.124573 5187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:56:55.142580 5187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:56:55.147399 5187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:56:55.147753 5187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:56:55.148949 5187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:56:55.150367 5187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:56:55.150480 5187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:55.150641 5187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:56:55.150715 5187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:55.185178 5187 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.1:46735
I20260928 14:56:55.185266 5314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.1:46735 every 8 connection(s)
I20260928 14:56:55.186479 5187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
I20260928 14:56:55.193775 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 5187
I20260928 14:56:55.194023 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/ts-0/wal/instance
I20260928 14:56:55.194864 31904 external_mini_cluster.cc:1428] Setting key 788b582fc7be0943af94bb488874434f
Sep 28 14:56:55 dist-test-slave-j968 krb5kdc[5081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607415, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.1@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:56:55.210775 5317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:55.189217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:49013 (local address 127.31.40.62:33541)
0928 14:56:55.189704 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:55.189712 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:55.190723 (+ 1011us) server_negotiation.cc:407] Connection header received
0928 14:56:55.192583 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:55.192588 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:55.192662 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:55.192767 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:55.194627 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:55.195777 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:55.197821 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:55.198215 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:55.198313 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:55.203632 (+ 5319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:55.203642 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:55.203651 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:55.203704 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:55.207054 (+ 3350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:55.208144 (+ 1090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:55.208149 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:55.208152 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:55.208285 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:55.208832 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:55.208835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:55.208837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:55.209193 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:55.209313 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:55.210326 (+ 1013us) server_negotiation.cc:299] Negotiation successful
0928 14:56:55.210513 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":232,"threads_started":1}
I20260928 14:56:55.212477 5318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:55.189824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:33541 (local address 127.31.40.1:49013)
0928 14:56:55.190550 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:55.190595 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:55.192313 (+ 1718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:55.192972 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:55.192981 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:55.193650 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:56:55.194387 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:55.194427 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:55.195989 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:55.196002 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:55.197672 (+ 1670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:55.197688 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:55.198469 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:55.198480 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:55.198858 (+ 378us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:55.199713 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:56:55.199756 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:56:55.203299 (+ 3543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:56:55.207232 (+ 3933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:55.207246 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:55.207269 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:55.208011 (+ 742us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:55.208411 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:55.208420 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:55.208426 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:55.208705 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:55.209328 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:56:55.209339 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:56:55.209847 (+ 508us) client_negotiation.cc:769] Sending connection context
0928 14:56:55.210184 (+ 337us) client_negotiation.cc:240] Negotiation successful
0928 14:56:55.210845 (+ 661us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":306,"threads_started":1}
I20260928 14:56:55.214447 5315 heartbeater.cc:347] Connected to a master server at 127.31.40.62:33541
I20260928 14:56:55.214875 5315 heartbeater.cc:464] Registering TS with master...
I20260928 14:56:55.215816 5315 heartbeater.cc:511] Master 127.31.40.62:33541 requested a full tablet report, sending...
I20260928 14:56:55.218788 5126 ts_manager.cc:194] Registered new tserver with Master: 91541ec1a80e4bfcbe441eed71dc00c0 (127.31.40.1:46735)
WARNING: no policy specified for kudu/127.31.40.2@KRBTEST.COM; defaulting to no policy
I20260928 14:56:55.220733 5126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.1@KRBTEST.COM'} at 127.31.40.1:49013
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab.
Entry for principal kudu/127.31.40.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab.
I20260928 14:56:55.247725 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.2:0
--local_ip_for_outbound_sockets=127.31.40.2
--webserver_interface=127.31.40.2
--webserver_port=0
--tserver_master_addrs=127.31.40.62:33541
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab
--principal=kudu/127.31.40.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:34117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:56:55.583566 5321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:56:55.583930 5321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:56:55.584100 5321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:56:55.591771 5321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:56:55.591998 5321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:56:55.592196 5321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.2
I20260928 14:56:55.602882 5321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:34117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab
--principal=kudu/127.31.40.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.40.2
--webserver_port=0
--tserver_master_addrs=127.31.40.62:33541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.2
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:56:55.604377 5321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:56:55.605748 5321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:56:55.619688 5326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:55.619635 5329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:55.619665 5327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:55.620370 5321 server_base.cc:1034] running on GCE node
I20260928 14:56:55.621061 5321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:56:55.622125 5321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:56:55.623375 5321 hybrid_clock.cc:648] HybridClock initialized: now 1790607415623321 us; error 46 us; skew 500 ppm
I20260928 14:56:55.623760 5321 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:56:55 dist-test-slave-j968 krb5kdc[5081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607415, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:56:55.627221 5321 init.cc:377] Logged in from keytab as kudu/127.31.40.2@KRBTEST.COM (short username kudu)
I20260928 14:56:55.628990 5321 webserver.cc:460] Webserver started at http://127.31.40.2:45483/ using document root <none> and password file <none>
I20260928 14:56:55.629784 5321 fs_manager.cc:359] Metadata directory not provided
I20260928 14:56:55.629927 5321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:56:55.630245 5321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:56:55.633481 5321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/ts-1/data/instance:
uuid: "002618cf555b46d280d7823314d12445"
format_stamp: "Formatted at 2026-09-28 14:56:55 on dist-test-slave-j968"
server_key: "b9e3357e2a01f99b58380c67748a4fb6"
server_key_iv: "2353a13e15838c1638b7c7b1ad833a47"
server_key_version: "encryptionkey@0"
I20260928 14:56:55.634373 5321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/ts-1/wal/instance:
uuid: "002618cf555b46d280d7823314d12445"
format_stamp: "Formatted at 2026-09-28 14:56:55 on dist-test-slave-j968"
server_key: "b9e3357e2a01f99b58380c67748a4fb6"
server_key_iv: "2353a13e15838c1638b7c7b1ad833a47"
server_key_version: "encryptionkey@0"
I20260928 14:56:55.639935 5321 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 14:56:55.643688 5336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:55.645299 5321 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:56:55.645519 5321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/ts-1/wal
uuid: "002618cf555b46d280d7823314d12445"
format_stamp: "Formatted at 2026-09-28 14:56:55 on dist-test-slave-j968"
server_key: "b9e3357e2a01f99b58380c67748a4fb6"
server_key_iv: "2353a13e15838c1638b7c7b1ad833a47"
server_key_version: "encryptionkey@0"
I20260928 14:56:55.645776 5321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:56:55.661128 5321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:56:55.665769 5321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:56:55.666143 5321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:56:55.667156 5321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:56:55.668597 5321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:56:55.668668 5321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:55.668831 5321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:56:55.668895 5321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:55.703131 5321 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.2:43809
I20260928 14:56:55.703240 5448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.2:43809 every 8 connection(s)
I20260928 14:56:55.704586 5321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
I20260928 14:56:55.709604 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 5321
I20260928 14:56:55.709815 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/ts-1/wal/instance
I20260928 14:56:55.710466 31904 external_mini_cluster.cc:1428] Setting key 93c91f54002bd3b17212264d5ea0659c
Sep 28 14:56:55 dist-test-slave-j968 krb5kdc[5081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607415, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.2@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:56:55.732357 5317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:55.707596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:46215 (local address 127.31.40.62:33541)
0928 14:56:55.707798 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:55.707802 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:55.709485 (+ 1683us) server_negotiation.cc:407] Connection header received
0928 14:56:55.712273 (+ 2788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:55.712279 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:55.712354 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:55.712478 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:55.715177 (+ 2699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:55.716033 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:55.719098 (+ 3065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:55.719554 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:55.719678 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:55.724061 (+ 4383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:55.724071 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:55.724079 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:55.724129 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:55.728069 (+ 3940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:55.729120 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:55.729130 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:55.729137 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:55.729311 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:55.729873 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:55.729880 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:55.729886 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:55.730311 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:55.730531 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:55.731782 (+ 1251us) server_negotiation.cc:299] Negotiation successful
0928 14:56:55.732097 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 14:56:55.734509 5451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:55.708181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:33541 (local address 127.31.40.2:46215)
0928 14:56:55.709222 (+ 1041us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:55.709297 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:55.711958 (+ 2661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:55.712775 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:55.712811 (+ 36us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:55.713888 (+ 1077us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:56:55.714926 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:55.714962 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:55.716223 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:55.716233 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:55.718884 (+ 2651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:55.718906 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:55.719835 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:55.719846 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:55.720267 (+ 421us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:55.721079 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:56:55.721107 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:56:55.723748 (+ 2641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:56:55.728268 (+ 4520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:55.728275 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:55.728291 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:55.728921 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:55.729470 (+ 549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:55.729474 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:55.729476 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:55.729690 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:55.730479 (+ 789us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:56:55.730486 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:56:55.731105 (+ 619us) client_negotiation.cc:769] Sending connection context
0928 14:56:55.731656 (+ 551us) client_negotiation.cc:240] Negotiation successful
0928 14:56:55.732512 (+ 856us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":331,"threads_started":1}
WARNING: no policy specified for kudu/127.31.40.3@KRBTEST.COM; defaulting to no policy
I20260928 14:56:55.736409 5449 heartbeater.cc:347] Connected to a master server at 127.31.40.62:33541
I20260928 14:56:55.736814 5449 heartbeater.cc:464] Registering TS with master...
I20260928 14:56:55.737730 5449 heartbeater.cc:511] Master 127.31.40.62:33541 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.3@KRBTEST.COM" created.
I20260928 14:56:55.739707 5126 ts_manager.cc:194] Registered new tserver with Master: 002618cf555b46d280d7823314d12445 (127.31.40.2:43809)
I20260928 14:56:55.740667 5126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.2@KRBTEST.COM'} at 127.31.40.2:46215
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab.
Entry for principal kudu/127.31.40.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab.
I20260928 14:56:55.768234 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.3:0
--local_ip_for_outbound_sockets=127.31.40.3
--webserver_interface=127.31.40.3
--webserver_port=0
--tserver_master_addrs=127.31.40.62:33541
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab
--principal=kudu/127.31.40.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:34117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:56:56.113478 5454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:56:56.113777 5454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:56:56.113947 5454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:56:56.121510 5454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:56:56.121635 5454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:56:56.121817 5454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.3
I20260928 14:56:56.132158 5454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:34117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab
--principal=kudu/127.31.40.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.40.3
--webserver_port=0
--tserver_master_addrs=127.31.40.62:33541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.3
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:56:56.134011 5454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:56:56.135955 5454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:56:56.150640 5459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:56.150731 5460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:56.150754 5462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:56.151489 5454 server_base.cc:1034] running on GCE node
I20260928 14:56:56.152228 5454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:56:56.153234 5454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:56:56.154466 5454 hybrid_clock.cc:648] HybridClock initialized: now 1790607416154402 us; error 47 us; skew 500 ppm
I20260928 14:56:56.154778 5454 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:56:56 dist-test-slave-j968 krb5kdc[5081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607416, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:56:56.158231 5454 init.cc:377] Logged in from keytab as kudu/127.31.40.3@KRBTEST.COM (short username kudu)
I20260928 14:56:56.159749 5454 webserver.cc:460] Webserver started at http://127.31.40.3:33091/ using document root <none> and password file <none>
I20260928 14:56:56.160530 5454 fs_manager.cc:359] Metadata directory not provided
I20260928 14:56:56.160632 5454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:56:56.160889 5454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:56:56.163933 5454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/ts-2/data/instance:
uuid: "ae5f94beb0454147a90e82a2420b1b50"
format_stamp: "Formatted at 2026-09-28 14:56:56 on dist-test-slave-j968"
server_key: "6290e0ffb94dd7e1371459c76cd0ca38"
server_key_iv: "c317f32066f21dd157b8080673c80e8b"
server_key_version: "encryptionkey@0"
I20260928 14:56:56.164718 5454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/ts-2/wal/instance:
uuid: "ae5f94beb0454147a90e82a2420b1b50"
format_stamp: "Formatted at 2026-09-28 14:56:56 on dist-test-slave-j968"
server_key: "6290e0ffb94dd7e1371459c76cd0ca38"
server_key_iv: "c317f32066f21dd157b8080673c80e8b"
server_key_version: "encryptionkey@0"
I20260928 14:56:56.170392 5454 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260928 14:56:56.174333 5469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:56.175976 5454 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:56:56.176194 5454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/ts-2/wal
uuid: "ae5f94beb0454147a90e82a2420b1b50"
format_stamp: "Formatted at 2026-09-28 14:56:56 on dist-test-slave-j968"
server_key: "6290e0ffb94dd7e1371459c76cd0ca38"
server_key_iv: "c317f32066f21dd157b8080673c80e8b"
server_key_version: "encryptionkey@0"
I20260928 14:56:56.176431 5454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:56:56.203562 5454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:56:56.208384 5454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:56:56.208684 5454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:56:56.209676 5454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:56:56.211125 5454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:56:56.211225 5454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:56.211380 5454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:56:56.211457 5454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:56.224921 5315 heartbeater.cc:503] Master 127.31.40.62:33541 was elected leader, sending a full tablet report...
I20260928 14:56:56.247506 5454 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.3:46637
I20260928 14:56:56.247557 5581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.3:46637 every 8 connection(s)
I20260928 14:56:56.248879 5454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
I20260928 14:56:56.251374 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 5454
I20260928 14:56:56.251652 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/minicluster-data/ts-2/wal/instance
I20260928 14:56:56.252454 31904 external_mini_cluster.cc:1428] Setting key 48bacad59367fdcb1d3e73ed46fae012
Sep 28 14:56:56 dist-test-slave-j968 krb5kdc[5081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607416, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.3@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:56:56.271461 5584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:56.251708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:51751 (local address 127.31.40.62:33541)
0928 14:56:56.252312 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:56.252320 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:56.253635 (+ 1315us) server_negotiation.cc:407] Connection header received
0928 14:56:56.256367 (+ 2732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:56.256371 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:56.256479 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:56.256587 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:56.258448 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:56.259345 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:56.261138 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:56.261606 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:56.261719 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:56.265397 (+ 3678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:56.265402 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:56.265405 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:56.265438 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:56.268549 (+ 3111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:56.269361 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:56.269365 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:56.269367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:56.269493 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:56.269877 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:56.269880 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:56.269882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:56.270200 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:56.270308 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:56.271039 (+ 731us) server_negotiation.cc:299] Negotiation successful
0928 14:56:56.271200 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":336,"threads_started":1}
I20260928 14:56:56.272555 5585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:56.252419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:33541 (local address 127.31.40.3:51751)
0928 14:56:56.253413 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:56.253490 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:56.255966 (+ 2476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:56.256821 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:56.256830 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:56.257531 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:56:56.258213 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:56.258235 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:56.259534 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:56.259538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:56.260984 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:56.260997 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:56.261845 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:56.261850 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:56.262101 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:56.262791 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:56:56.262814 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:56:56.265169 (+ 2355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:56:56.268699 (+ 3530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:56.268708 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:56.268726 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:56.269230 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:56.269592 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:56.269595 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:56.269597 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:56.269780 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:56.270314 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:56:56.270319 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:56:56.270643 (+ 324us) client_negotiation.cc:769] Sending connection context
0928 14:56:56.270864 (+ 221us) client_negotiation.cc:240] Negotiation successful
0928 14:56:56.271260 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":401,"threads_started":1}
I20260928 14:56:56.274027 5582 heartbeater.cc:347] Connected to a master server at 127.31.40.62:33541
I20260928 14:56:56.274340 5582 heartbeater.cc:464] Registering TS with master...
I20260928 14:56:56.275156 5582 heartbeater.cc:511] Master 127.31.40.62:33541 requested a full tablet report, sending...
I20260928 14:56:56.276711 5126 ts_manager.cc:194] Registered new tserver with Master: ae5f94beb0454147a90e82a2420b1b50 (127.31.40.3:46637)
I20260928 14:56:56.277518 5126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.3@KRBTEST.COM'} at 127.31.40.3:51751
I20260928 14:56:56.281186 31904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/krb5kdc/test-admin.keytab.
Sep 28 14:56:56 dist-test-slave-j968 krb5kdc[5081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:56:56.311105 31904 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 14:56:56.317288 5590 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:56:56.314372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57724 (local address 127.31.40.1:46735)
0928 14:56:56.315088 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:56.315096 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:56.315126 (+ 30us) server_negotiation.cc:407] Connection header received
0928 14:56:56.315194 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:56.315200 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:56.315448 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:56.315598 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:56.316865 (+ 1267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57724: BlockingRecv error: recv got EOF from 127.0.0.1:57724 (error 108)
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":364,"threads_started":1}
W20260928 14:56:56.317298 31904 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/krb5kdc/krb5cc' of type FILE
Sep 28 14:56:56 dist-test-slave-j968 krb5kdc[5081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:56:56.319392 31904 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 14:56:56 dist-test-slave-j968 krb5kdc[5081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:56:56.321056 31904 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 14:56:56 dist-test-slave-j968 krb5kdc[5081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.1@KRBTEST.COM
I20260928 14:56:56.338922 5590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:56.323036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57738 (local address 127.31.40.1:46735)
0928 14:56:56.323273 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:56.323281 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:56.323510 (+ 229us) server_negotiation.cc:407] Connection header received
0928 14:56:56.323748 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:56.323753 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:56.323817 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:56.323955 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:56.325104 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:56.326120 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:56.327416 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:56.328099 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:56.328238 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:56.332519 (+ 4281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:56.332535 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:56.332552 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:56.332605 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:56.336158 (+ 3553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:56.336913 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:56.336921 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:56.336923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:56.337093 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:56.337464 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:56.337467 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:56.337469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:56.337820 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:56.338073 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:56.338338 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 14:56:56.338648 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260928 14:56:56.345992 5590 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:56:56.343709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57754 (local address 127.31.40.1:46735)
0928 14:56:56.344143 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:56.344151 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:56.344380 (+ 229us) server_negotiation.cc:407] Connection header received
0928 14:56:56.344645 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:56.344650 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:56.344725 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:56.344846 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:56.345721 (+ 875us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57754: BlockingRecv error: recv got EOF from 127.0.0.1:57754 (error 108)
Metrics: {"server-negotiator.queue_time_us":138}
W20260928 14:56:56.346331 31904 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/krb5kdc/krb5cc' of type FILE
Sep 28 14:56:56 dist-test-slave-j968 krb5kdc[5081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:56:56.348021 31904 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 14:56:56 dist-test-slave-j968 krb5kdc[5081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:56:56.349582 31904 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 14:56:56 dist-test-slave-j968 krb5kdc[5081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.1@KRBTEST.COM
I20260928 14:56:56.364651 5590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:56.351285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57764 (local address 127.31.40.1:46735)
0928 14:56:56.351561 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:56.351569 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:56.351813 (+ 244us) server_negotiation.cc:407] Connection header received
0928 14:56:56.352063 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:56.352067 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:56.352129 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:56.352235 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:56.353192 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:56.353908 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:56.355164 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:56.355583 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:56.355717 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:56.359385 (+ 3668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:56.359390 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:56.359393 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:56.359428 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:56.362074 (+ 2646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:56.362765 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:56.362778 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:56.362780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:56.362900 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:56.363272 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:56.363278 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:56.363281 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:56.363554 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:56.363841 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:56.364155 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 14:56:56.364366 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260928 14:56:56.369217 5590 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:56:56.367301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57774 (local address 127.31.40.1:46735)
0928 14:56:56.367444 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:56.367448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:56.367785 (+ 337us) server_negotiation.cc:407] Connection header received
0928 14:56:56.367959 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:56.367963 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:56.368058 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:56.368134 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:56.368961 (+ 827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57774: BlockingRecv error: recv got EOF from 127.0.0.1:57774 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260928 14:56:56.369616 31904 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607365362981-31904-0/krb5kdc/krb5cc' of type FILE
Sep 28 14:56:56 dist-test-slave-j968 krb5kdc[5081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:56:56.371173 31904 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 14:56:56 dist-test-slave-j968 krb5kdc[5081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:56:56.372649 31904 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 14:56:56 dist-test-slave-j968 krb5kdc[5081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.1@KRBTEST.COM
I20260928 14:56:56.387778 5590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:56.374182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57788 (local address 127.31.40.1:46735)
0928 14:56:56.374407 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:56.374414 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:56.374600 (+ 186us) server_negotiation.cc:407] Connection header received
0928 14:56:56.374774 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:56.374779 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:56.374835 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:56.374932 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:56.375895 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:56.376611 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:56.377860 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:56.378314 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:56.378442 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:56.382363 (+ 3921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:56.382368 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:56.382371 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:56.382402 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:56.385143 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:56.385824 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:56.385829 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:56.385831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:56.386031 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:56.386558 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:56.386562 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:56.386564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:56.386857 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:56.387106 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:56.387333 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 14:56:56.387506 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 14:56:56.389585 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 5187
I20260928 14:56:56.402457 5590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:56.390494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57790 (local address 127.31.40.1:46735)
0928 14:56:56.390720 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:56.390727 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:56.390978 (+ 251us) server_negotiation.cc:407] Connection header received
0928 14:56:56.391082 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:56.391085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:56.391142 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:56.391219 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:56.392386 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:56.393140 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:56.394487 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:56.394947 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:56.395047 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:56.397136 (+ 2089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:56.397141 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:56.397144 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:56.397174 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:56.399632 (+ 2458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:56.400385 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:56.400391 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:56.400395 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:56.400536 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:56.400928 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:56.400933 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:56.400941 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:56.401259 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:56.401590 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:56.401899 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 14:56:56.402124 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 14:56:56.403261 5310 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:56:56.487922 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 5187
I20260928 14:56:56.505656 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 5321
Sep 28 14:56:56 dist-test-slave-j968 krb5kdc[5081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.2@KRBTEST.COM
I20260928 14:56:56.522621 5603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:56.506616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53062 (local address 127.31.40.2:43809)
0928 14:56:56.507232 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:56.507240 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:56.507268 (+ 28us) server_negotiation.cc:407] Connection header received
0928 14:56:56.507330 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:56.507337 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:56.507556 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:56.507686 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:56.508738 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:56.509838 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:56.511133 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:56.511728 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:56.511831 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:56.516090 (+ 4259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:56.516105 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:56.516123 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:56.516165 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:56.519399 (+ 3234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:56.520442 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:56.520449 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:56.520451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:56.520614 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:56.521132 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:56.521136 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:56.521138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:56.521508 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:56.521735 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:56.522094 (+ 359us) server_negotiation.cc:299] Negotiation successful
0928 14:56:56.522294 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":292,"threads_started":1}
I20260928 14:56:56.524112 5444 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:56:56.609898 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 5321
I20260928 14:56:56.627460 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 5454
Sep 28 14:56:56 dist-test-slave-j968 krb5kdc[5081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.3@KRBTEST.COM
I20260928 14:56:56.644726 5605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:56.628405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46200 (local address 127.31.40.3:46637)
0928 14:56:56.629038 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:56.629045 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:56.629075 (+ 30us) server_negotiation.cc:407] Connection header received
0928 14:56:56.629138 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:56.629145 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:56.629366 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:56.629500 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:56.630532 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:56.631598 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:56.633108 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:56.633693 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:56.633859 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:56.638253 (+ 4394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:56.638269 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:56.638287 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:56.638336 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:56.641496 (+ 3160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:56.642322 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:56.642331 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:56.642333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:56.642505 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:56.642931 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:56.642937 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:56.642940 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:56.643338 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:56.643619 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:56.644056 (+ 437us) server_negotiation.cc:299] Negotiation successful
0928 14:56:56.644369 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":310,"threads_started":1}
I20260928 14:56:56.646111 5577 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:56:56.730436 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 5454
I20260928 14:56:56.747988 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 5094
I20260928 14:56:56.749522 5156 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:56:56.856478 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 5094
2026-09-28T14:56:56Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4127 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:56:56 dist-test-slave-j968 krb5kdc[5610](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:56:56 dist-test-slave-j968 krb5kdc[5610](info): set up 2 sockets
Sep 28 14:56:56 dist-test-slave-j968 krb5kdc[5610](info): commencing operation
krb5kdc: starting...
W20260928 14:56:57.961714 31904 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Sep 28 14:56:58 dist-test-slave-j968 krb5kdc[5610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:56:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:56:58Z Disabled control of system clock
WARNING: no policy specified for oryx/127.31.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/krb5kdc/oryx_127.31.40.62.keytab.
Entry for principal oryx/127.31.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/krb5kdc/oryx_127.31.40.62.keytab.
I20260928 14:56:58.185106 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:32909
--webserver_interface=127.31.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/krb5kdc/oryx_127.31.40.62.keytab
--principal=oryx/127.31.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:32961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.40.62:32909
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:56:58.517278 5623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:56:58.517624 5623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:56:58.517788 5623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:56:58.525082 5623 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:56:58.525235 5623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:56:58.525352 5623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:56:58.525451 5623 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:56:58.525534 5623 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:56:58.535372 5623 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:32961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.40.62:32909
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/krb5kdc/oryx_127.31.40.62.keytab
--principal=oryx/127.31.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:32909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:56:58.536831 5623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:56:58.538213 5623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:56:58.550953 5628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:58.550892 5629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:58.550943 5631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:58.551857 5623 server_base.cc:1034] running on GCE node
I20260928 14:56:58.552781 5623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:56:58.554422 5623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:56:58.555703 5623 hybrid_clock.cc:648] HybridClock initialized: now 1790607418555653 us; error 72 us; skew 500 ppm
I20260928 14:56:58.556154 5623 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:56:58 dist-test-slave-j968 krb5kdc[5610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607418, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:56:58.559913 5623 init.cc:377] Logged in from keytab as oryx/127.31.40.62@KRBTEST.COM (short username oryx)
I20260928 14:56:58.561691 5623 webserver.cc:460] Webserver started at http://127.31.40.62:34401/ using document root <none> and password file <none>
I20260928 14:56:58.562531 5623 fs_manager.cc:359] Metadata directory not provided
I20260928 14:56:58.562698 5623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:56:58.563047 5623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:56:58.566608 5623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-0/data/instance:
uuid: "40f51af6fc714868a4c4df423f897b6c"
format_stamp: "Formatted at 2026-09-28 14:56:58 on dist-test-slave-j968"
server_key: "fc1d22f7a96aab0bb70e8ace69c28b2b"
server_key_iv: "813799022a2693efb647b922496e0f08"
server_key_version: "encryptionkey@0"
I20260928 14:56:58.567600 5623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-0/wal/instance:
uuid: "40f51af6fc714868a4c4df423f897b6c"
format_stamp: "Formatted at 2026-09-28 14:56:58 on dist-test-slave-j968"
server_key: "fc1d22f7a96aab0bb70e8ace69c28b2b"
server_key_iv: "813799022a2693efb647b922496e0f08"
server_key_version: "encryptionkey@0"
I20260928 14:56:58.574218 5623 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260928 14:56:58.578891 5638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:58.580567 5623 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:56:58.580786 5623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-0/wal
uuid: "40f51af6fc714868a4c4df423f897b6c"
format_stamp: "Formatted at 2026-09-28 14:56:58 on dist-test-slave-j968"
server_key: "fc1d22f7a96aab0bb70e8ace69c28b2b"
server_key_iv: "813799022a2693efb647b922496e0f08"
server_key_version: "encryptionkey@0"
I20260928 14:56:58.581030 5623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:56:58.602691 5623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:56:58.607431 5623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:56:58.607759 5623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:56:58.629487 5623 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.62:32909
I20260928 14:56:58.629521 5689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.62:32909 every 8 connection(s)
I20260928 14:56:58.631120 5623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
I20260928 14:56:58.635434 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:56:58.636621 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 5623
I20260928 14:56:58.636860 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-0/wal/instance
I20260928 14:56:58.637742 31904 external_mini_cluster.cc:1428] Setting key d63708dd834081219d24a0e443e8a101
I20260928 14:56:58.651125 5690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40f51af6fc714868a4c4df423f897b6c: Bootstrap starting.
Sep 28 14:56:58 dist-test-slave-j968 krb5kdc[5610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM
I20260928 14:56:58.654795 5690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40f51af6fc714868a4c4df423f897b6c: Neither blocks nor log segments found. Creating new log.
I20260928 14:56:58.655973 5690 log.cc:824] T 00000000000000000000000000000000 P 40f51af6fc714868a4c4df423f897b6c: Log is configured to *not* fsync() on all Append() calls
I20260928 14:56:58.659912 5690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40f51af6fc714868a4c4df423f897b6c: No bootstrap required, opened a new log
I20260928 14:56:58.665285 5693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:58.640098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59318 (local address 127.31.40.62:32909)
0928 14:56:58.641011 (+ 913us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:58.641032 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:58.641097 (+ 65us) server_negotiation.cc:407] Connection header received
0928 14:56:58.643801 (+ 2704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:58.643841 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:58.644467 (+ 626us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:58.645032 (+ 565us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:58.646681 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:58.648298 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:58.649752 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:58.650545 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:58.650787 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:58.655440 (+ 4653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:58.655449 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:58.655472 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:58.655517 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:58.659918 (+ 4401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:58.660780 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:58.660789 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:58.660800 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:58.661036 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:58.661435 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:58.661441 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:58.661445 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:58.662324 (+ 879us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:58.662691 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:58.663084 (+ 393us) server_negotiation.cc:299] Negotiation successful
0928 14:56:58.663626 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":411,"threads_started":1}
I20260928 14:56:58.671350 5690 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40f51af6fc714868a4c4df423f897b6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40f51af6fc714868a4c4df423f897b6c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32909 } }
I20260928 14:56:58.671748 5690 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40f51af6fc714868a4c4df423f897b6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:56:58.671854 5690 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40f51af6fc714868a4c4df423f897b6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40f51af6fc714868a4c4df423f897b6c, State: Initialized, Role: FOLLOWER
I20260928 14:56:58.672488 5690 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40f51af6fc714868a4c4df423f897b6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40f51af6fc714868a4c4df423f897b6c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32909 } }
I20260928 14:56:58.672721 5690 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40f51af6fc714868a4c4df423f897b6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:56:58.672845 5690 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40f51af6fc714868a4c4df423f897b6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:56:58.673014 5690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40f51af6fc714868a4c4df423f897b6c [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:56:58.675321 5690 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40f51af6fc714868a4c4df423f897b6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40f51af6fc714868a4c4df423f897b6c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32909 } }
I20260928 14:56:58.675827 5690 leader_election.cc:302] T 00000000000000000000000000000000 P 40f51af6fc714868a4c4df423f897b6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40f51af6fc714868a4c4df423f897b6c; no voters:
I20260928 14:56:58.676532 5690 leader_election.cc:288] T 00000000000000000000000000000000 P 40f51af6fc714868a4c4df423f897b6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:56:58.676759 5695 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40f51af6fc714868a4c4df423f897b6c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:56:58.677614 5695 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40f51af6fc714868a4c4df423f897b6c [term 1 LEADER]: Becoming Leader. State: Replica: 40f51af6fc714868a4c4df423f897b6c, State: Running, Role: LEADER
I20260928 14:56:58.678280 5695 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40f51af6fc714868a4c4df423f897b6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40f51af6fc714868a4c4df423f897b6c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32909 } }
I20260928 14:56:58.679020 5690 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40f51af6fc714868a4c4df423f897b6c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:56:58.681512 5696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40f51af6fc714868a4c4df423f897b6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40f51af6fc714868a4c4df423f897b6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40f51af6fc714868a4c4df423f897b6c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32909 } } }
I20260928 14:56:58.681947 5696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40f51af6fc714868a4c4df423f897b6c [sys.catalog]: This master's current role is: LEADER
I20260928 14:56:58.681773 5697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40f51af6fc714868a4c4df423f897b6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40f51af6fc714868a4c4df423f897b6c. Latest consensus state: current_term: 1 leader_uuid: "40f51af6fc714868a4c4df423f897b6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40f51af6fc714868a4c4df423f897b6c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32909 } } }
I20260928 14:56:58.682227 5697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40f51af6fc714868a4c4df423f897b6c [sys.catalog]: This master's current role is: LEADER
I20260928 14:56:58.683274 5704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:56:58.689462 5704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:56:58.699086 5704 catalog_manager.cc:1347] Generated new cluster ID: 913af42657514f2aa51a685dba5d7ba1
I20260928 14:56:58.699218 5704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:56:58.717603 5704 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:56:58.719527 5704 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:56:58.728885 5704 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40f51af6fc714868a4c4df423f897b6c: Generated new TSK 0
I20260928 14:56:58.730103 5704 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.31.40.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.40.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.40.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/krb5kdc/oryx_127.31.40.1.keytab.
Entry for principal oryx/127.31.40.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/krb5kdc/oryx_127.31.40.1.keytab.
I20260928 14:56:58.791055 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.1:0
--local_ip_for_outbound_sockets=127.31.40.1
--webserver_interface=127.31.40.1
--webserver_port=0
--tserver_master_addrs=127.31.40.62:32909
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/krb5kdc/oryx_127.31.40.1.keytab
--principal=oryx/127.31.40.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:32961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:56:59.135025 5716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:56:59.135322 5716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:56:59.135504 5716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:56:59.143190 5716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:56:59.143309 5716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:56:59.143543 5716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.1
I20260928 14:56:59.153724 5716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:32961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/krb5kdc/oryx_127.31.40.1.keytab
--principal=oryx/127.31.40.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.40.1
--webserver_port=0
--tserver_master_addrs=127.31.40.62:32909
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.1
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:56:59.155164 5716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:56:59.156627 5716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:56:59.170974 5721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:59.170953 5722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:59.171061 5724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:59.171731 5716 server_base.cc:1034] running on GCE node
I20260928 14:56:59.172513 5716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:56:59.173549 5716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:56:59.174749 5716 hybrid_clock.cc:648] HybridClock initialized: now 1790607419174705 us; error 33 us; skew 500 ppm
I20260928 14:56:59.175087 5716 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:56:59 dist-test-slave-j968 krb5kdc[5610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607419, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.40.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:56:59.178269 5716 init.cc:377] Logged in from keytab as oryx/127.31.40.1@KRBTEST.COM (short username oryx)
I20260928 14:56:59.179852 5716 webserver.cc:460] Webserver started at http://127.31.40.1:40027/ using document root <none> and password file <none>
I20260928 14:56:59.180642 5716 fs_manager.cc:359] Metadata directory not provided
I20260928 14:56:59.180751 5716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:56:59.181003 5716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:56:59.184336 5716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-0/data/instance:
uuid: "50bbf97992ac4d92a0ce12b7b1be3bbe"
format_stamp: "Formatted at 2026-09-28 14:56:59 on dist-test-slave-j968"
server_key: "26442481750400b0e779d3b7da95122f"
server_key_iv: "896fbc2040a586b1d715c9dcfb3287ad"
server_key_version: "encryptionkey@0"
I20260928 14:56:59.185160 5716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-0/wal/instance:
uuid: "50bbf97992ac4d92a0ce12b7b1be3bbe"
format_stamp: "Formatted at 2026-09-28 14:56:59 on dist-test-slave-j968"
server_key: "26442481750400b0e779d3b7da95122f"
server_key_iv: "896fbc2040a586b1d715c9dcfb3287ad"
server_key_version: "encryptionkey@0"
I20260928 14:56:59.190423 5716 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260928 14:56:59.194096 5731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:59.195632 5716 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:56:59.195768 5716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-0/wal
uuid: "50bbf97992ac4d92a0ce12b7b1be3bbe"
format_stamp: "Formatted at 2026-09-28 14:56:59 on dist-test-slave-j968"
server_key: "26442481750400b0e779d3b7da95122f"
server_key_iv: "896fbc2040a586b1d715c9dcfb3287ad"
server_key_version: "encryptionkey@0"
I20260928 14:56:59.196020 5716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:56:59.215670 5716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:56:59.220350 5716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:56:59.220605 5716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:56:59.221529 5716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:56:59.222890 5716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:56:59.222966 5716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:59.223052 5716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:56:59.223097 5716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:59.257267 5716 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.1:35061
I20260928 14:56:59.257369 5843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.1:35061 every 8 connection(s)
I20260928 14:56:59.258590 5716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
I20260928 14:56:59.264428 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 5716
I20260928 14:56:59.264653 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-0/wal/instance
I20260928 14:56:59.265305 31904 external_mini_cluster.cc:1428] Setting key 0c6e0eab5f2e2a9acd53f99df0bf3805
Sep 28 14:56:59 dist-test-slave-j968 krb5kdc[5610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607419, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.40.1@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM
I20260928 14:56:59.280838 5846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:59.261292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:47311 (local address 127.31.40.62:32909)
0928 14:56:59.261774 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:59.261779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:59.262631 (+ 852us) server_negotiation.cc:407] Connection header received
0928 14:56:59.264977 (+ 2346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:59.264983 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:59.265083 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:59.265206 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:59.266831 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:59.267608 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:59.269308 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:59.269712 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:59.269857 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:59.273620 (+ 3763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:59.273629 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:59.273643 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:59.273692 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:59.277402 (+ 3710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:59.278152 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:59.278161 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:59.278167 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:59.278313 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:59.278733 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:59.278740 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:59.278746 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:59.279146 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:59.279306 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:59.280106 (+ 800us) server_negotiation.cc:299] Negotiation successful
0928 14:56:59.280362 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":274,"threads_started":1}
I20260928 14:56:59.281687 5847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:59.261690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:32909 (local address 127.31.40.1:47311)
0928 14:56:59.262459 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:59.262507 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:59.264646 (+ 2139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:59.265480 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:59.265490 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:59.266026 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:56:59.266656 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:59.266673 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:59.267766 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:59.267771 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:59.269150 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:59.269163 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:59.269991 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:59.270000 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:59.270233 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:59.270921 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:56:59.270950 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:56:59.273375 (+ 2425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:56:59.277553 (+ 4178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:59.277560 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:59.277573 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:59.278012 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:59.278431 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:59.278434 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:59.278438 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:59.278615 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:59.279292 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:56:59.279298 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:56:59.279666 (+ 368us) client_negotiation.cc:769] Sending connection context
0928 14:56:59.279978 (+ 312us) client_negotiation.cc:240] Negotiation successful
0928 14:56:59.280478 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":295,"threads_started":1}
I20260928 14:56:59.283463 5844 heartbeater.cc:347] Connected to a master server at 127.31.40.62:32909
I20260928 14:56:59.283880 5844 heartbeater.cc:464] Registering TS with master...
I20260928 14:56:59.284893 5844 heartbeater.cc:511] Master 127.31.40.62:32909 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.31.40.2@KRBTEST.COM; defaulting to no policy
I20260928 14:56:59.287295 5655 ts_manager.cc:194] Registered new tserver with Master: 50bbf97992ac4d92a0ce12b7b1be3bbe (127.31.40.1:35061)
I20260928 14:56:59.288848 5655 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.40.1@KRBTEST.COM'} at 127.31.40.1:47311
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.40.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.40.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/krb5kdc/oryx_127.31.40.2.keytab.
Entry for principal oryx/127.31.40.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/krb5kdc/oryx_127.31.40.2.keytab.
I20260928 14:56:59.315307 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.2:0
--local_ip_for_outbound_sockets=127.31.40.2
--webserver_interface=127.31.40.2
--webserver_port=0
--tserver_master_addrs=127.31.40.62:32909
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/krb5kdc/oryx_127.31.40.2.keytab
--principal=oryx/127.31.40.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:32961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:56:59.653338 5850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:56:59.653699 5850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:56:59.653870 5850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:56:59.661407 5850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:56:59.661582 5850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:56:59.661770 5850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.2
I20260928 14:56:59.671958 5850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:32961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/krb5kdc/oryx_127.31.40.2.keytab
--principal=oryx/127.31.40.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.40.2
--webserver_port=0
--tserver_master_addrs=127.31.40.62:32909
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.2
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:56:59.673377 5850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:56:59.674731 5850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:56:59.688642 5855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:59.688668 5858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:59.688591 5856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:59.689596 5850 server_base.cc:1034] running on GCE node
I20260928 14:56:59.690481 5850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:56:59.691474 5850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:56:59.692706 5850 hybrid_clock.cc:648] HybridClock initialized: now 1790607419692651 us; error 48 us; skew 500 ppm
I20260928 14:56:59.693099 5850 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:56:59 dist-test-slave-j968 krb5kdc[5610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607419, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.40.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:56:59.703030 5850 init.cc:377] Logged in from keytab as oryx/127.31.40.2@KRBTEST.COM (short username oryx)
I20260928 14:56:59.704793 5850 webserver.cc:460] Webserver started at http://127.31.40.2:41141/ using document root <none> and password file <none>
I20260928 14:56:59.705637 5850 fs_manager.cc:359] Metadata directory not provided
I20260928 14:56:59.705840 5850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:56:59.706185 5850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:56:59.709273 5850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-1/data/instance:
uuid: "1484aa0f119348a1af94cea1a988d0f2"
format_stamp: "Formatted at 2026-09-28 14:56:59 on dist-test-slave-j968"
server_key: "dfa0b50c70221f6722dddc8f540b7a74"
server_key_iv: "e46fa74b10ab334af7fb216ef7a49bf2"
server_key_version: "encryptionkey@0"
I20260928 14:56:59.710089 5850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-1/wal/instance:
uuid: "1484aa0f119348a1af94cea1a988d0f2"
format_stamp: "Formatted at 2026-09-28 14:56:59 on dist-test-slave-j968"
server_key: "dfa0b50c70221f6722dddc8f540b7a74"
server_key_iv: "e46fa74b10ab334af7fb216ef7a49bf2"
server_key_version: "encryptionkey@0"
I20260928 14:56:59.715500 5850 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 14:56:59.719836 5865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:59.721478 5850 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 14:56:59.721678 5850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-1/wal
uuid: "1484aa0f119348a1af94cea1a988d0f2"
format_stamp: "Formatted at 2026-09-28 14:56:59 on dist-test-slave-j968"
server_key: "dfa0b50c70221f6722dddc8f540b7a74"
server_key_iv: "e46fa74b10ab334af7fb216ef7a49bf2"
server_key_version: "encryptionkey@0"
I20260928 14:56:59.721921 5850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:56:59.759999 5850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:56:59.765461 5850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:56:59.765810 5850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:56:59.766980 5850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:56:59.768603 5850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:56:59.768724 5850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:59.768904 5850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:56:59.769001 5850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:59.804162 5850 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.2:33025
I20260928 14:56:59.804219 5977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.2:33025 every 8 connection(s)
I20260928 14:56:59.805569 5850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
I20260928 14:56:59.808697 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 5850
I20260928 14:56:59.808890 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-1/wal/instance
I20260928 14:56:59.809463 31904 external_mini_cluster.cc:1428] Setting key f58a9f265a08354d08f7f6a57e21505e
Sep 28 14:56:59 dist-test-slave-j968 krb5kdc[5610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607419, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.40.2@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM
I20260928 14:56:59.830312 5980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:59.808612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:46401 (local address 127.31.40.62:32909)
0928 14:56:59.809289 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:59.809294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:59.810328 (+ 1034us) server_negotiation.cc:407] Connection header received
0928 14:56:59.812595 (+ 2267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:59.812601 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:59.812706 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:59.812836 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:59.814518 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:59.815347 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:59.817386 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:59.817809 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:59.817913 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:59.822353 (+ 4440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:59.822362 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:59.822370 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:59.822417 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:59.826217 (+ 3800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:59.827269 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:59.827278 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:59.827285 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:59.827462 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:59.828045 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:59.828053 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:59.828057 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:59.828468 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:59.828649 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:59.829567 (+ 918us) server_negotiation.cc:299] Negotiation successful
0928 14:56:59.829844 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":450,"threads_started":1}
I20260928 14:56:59.831229 5981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:59.809259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:32909 (local address 127.31.40.2:46401)
0928 14:56:59.810145 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:59.810190 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:59.812288 (+ 2098us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:59.813110 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:59.813119 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:59.813675 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:56:59.814319 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:59.814338 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:59.815527 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:59.815540 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:59.817208 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:59.817228 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:59.818198 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:59.818207 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:59.818564 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:59.819472 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:56:59.819522 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:56:59.822080 (+ 2558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:56:59.826419 (+ 4339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:59.826428 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:59.826444 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:59.827118 (+ 674us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:59.827627 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:59.827634 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:59.827638 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:59.827849 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:59.828635 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:56:59.828643 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:56:59.829134 (+ 491us) client_negotiation.cc:769] Sending connection context
0928 14:56:59.829424 (+ 290us) client_negotiation.cc:240] Negotiation successful
0928 14:56:59.829895 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":405,"threads_started":1}
WARNING: no policy specified for oryx/127.31.40.3@KRBTEST.COM; defaulting to no policy
I20260928 14:56:59.833124 5978 heartbeater.cc:347] Connected to a master server at 127.31.40.62:32909
I20260928 14:56:59.833438 5978 heartbeater.cc:464] Registering TS with master...
I20260928 14:56:59.834424 5978 heartbeater.cc:511] Master 127.31.40.62:32909 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.40.3@KRBTEST.COM" created.
I20260928 14:56:59.836320 5655 ts_manager.cc:194] Registered new tserver with Master: 1484aa0f119348a1af94cea1a988d0f2 (127.31.40.2:33025)
I20260928 14:56:59.837239 5655 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.40.2@KRBTEST.COM'} at 127.31.40.2:46401
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.40.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/krb5kdc/oryx_127.31.40.3.keytab.
Entry for principal oryx/127.31.40.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/krb5kdc/oryx_127.31.40.3.keytab.
I20260928 14:56:59.864409 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.3:0
--local_ip_for_outbound_sockets=127.31.40.3
--webserver_interface=127.31.40.3
--webserver_port=0
--tserver_master_addrs=127.31.40.62:32909
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/krb5kdc/oryx_127.31.40.3.keytab
--principal=oryx/127.31.40.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:32961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:57:00.201267 5984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:00.201609 5984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:00.201781 5984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:00.209196 5984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:00.209388 5984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:00.209585 5984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.3
I20260928 14:57:00.220399 5984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:32961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/krb5kdc/oryx_127.31.40.3.keytab
--principal=oryx/127.31.40.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.40.3
--webserver_port=0
--tserver_master_addrs=127.31.40.62:32909
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.3
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:57:00.221805 5984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:00.223248 5984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:00.237460 5990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:00.237541 5992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:00.237843 5989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:00.239310 5984 server_base.cc:1034] running on GCE node
I20260928 14:57:00.240017 5984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:00.240988 5984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:00.242194 5984 hybrid_clock.cc:648] HybridClock initialized: now 1790607420242151 us; error 38 us; skew 500 ppm
I20260928 14:57:00.242532 5984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:57:00 dist-test-slave-j968 krb5kdc[5610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607420, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.40.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:00.245987 5984 init.cc:377] Logged in from keytab as oryx/127.31.40.3@KRBTEST.COM (short username oryx)
I20260928 14:57:00.247563 5984 webserver.cc:460] Webserver started at http://127.31.40.3:38755/ using document root <none> and password file <none>
I20260928 14:57:00.248323 5984 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:00.248426 5984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:00.248679 5984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:00.251590 5984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-2/data/instance:
uuid: "f2f0f28f0299472386abafe15704df7b"
format_stamp: "Formatted at 2026-09-28 14:57:00 on dist-test-slave-j968"
server_key: "2feb37b4657d920bbb6dd87a78119f3d"
server_key_iv: "8a57bdf247cd5a31be088d8bf64f6dfc"
server_key_version: "encryptionkey@0"
I20260928 14:57:00.252408 5984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-2/wal/instance:
uuid: "f2f0f28f0299472386abafe15704df7b"
format_stamp: "Formatted at 2026-09-28 14:57:00 on dist-test-slave-j968"
server_key: "2feb37b4657d920bbb6dd87a78119f3d"
server_key_iv: "8a57bdf247cd5a31be088d8bf64f6dfc"
server_key_version: "encryptionkey@0"
I20260928 14:57:00.258005 5984 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 14:57:00.262550 5999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:00.264052 5984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:57:00.264276 5984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-2/wal
uuid: "f2f0f28f0299472386abafe15704df7b"
format_stamp: "Formatted at 2026-09-28 14:57:00 on dist-test-slave-j968"
server_key: "2feb37b4657d920bbb6dd87a78119f3d"
server_key_iv: "8a57bdf247cd5a31be088d8bf64f6dfc"
server_key_version: "encryptionkey@0"
I20260928 14:57:00.264484 5984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:00.280416 5984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:00.284761 5984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:00.285072 5984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:00.286131 5984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:57:00.287503 5984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:57:00.287606 5984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:00.287758 5984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:57:00.287839 5984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:00.292085 5844 heartbeater.cc:503] Master 127.31.40.62:32909 was elected leader, sending a full tablet report...
I20260928 14:57:00.322213 5984 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.3:34255
I20260928 14:57:00.322306 6111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.3:34255 every 8 connection(s)
I20260928 14:57:00.323593 5984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
I20260928 14:57:00.326119 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 5984
I20260928 14:57:00.326336 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-2/wal/instance
I20260928 14:57:00.327055 31904 external_mini_cluster.cc:1428] Setting key 05c11d9e4f57b8219147f250523bb517
Sep 28 14:57:00 dist-test-slave-j968 krb5kdc[5610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607420, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.40.3@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM
I20260928 14:57:00.346763 5980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:00.326951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:51283 (local address 127.31.40.62:32909)
0928 14:57:00.327181 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:00.327189 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:00.328789 (+ 1600us) server_negotiation.cc:407] Connection header received
0928 14:57:00.330940 (+ 2151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:00.330945 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:00.331048 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:00.331180 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:00.333055 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:00.333796 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:00.335394 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:00.335819 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:00.335976 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:00.339795 (+ 3819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:00.339801 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:00.339805 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:00.339842 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:00.343545 (+ 3703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:00.344444 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:00.344448 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:00.344452 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:00.344590 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:00.345032 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:00.345036 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:00.345038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:00.345346 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:00.345483 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:00.346290 (+ 807us) server_negotiation.cc:299] Negotiation successful
0928 14:57:00.346487 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 14:57:00.348150 6114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:00.327590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:32909 (local address 127.31.40.3:51283)
0928 14:57:00.328604 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:00.328670 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:00.330669 (+ 1999us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:00.331447 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:00.331456 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:00.332222 (+ 766us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:00.332871 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:00.332891 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:00.333967 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:00.333971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:00.335249 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:00.335261 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:00.336091 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:00.336099 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:00.336346 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:00.337131 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:00.337155 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:00.339542 (+ 2387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:00.343681 (+ 4139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:00.343689 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:00.343704 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:00.344277 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:00.344693 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:00.344697 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:00.344699 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:00.344919 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:00.345507 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:00.345514 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:00.345913 (+ 399us) client_negotiation.cc:769] Sending connection context
0928 14:57:00.346184 (+ 271us) client_negotiation.cc:240] Negotiation successful
0928 14:57:00.346642 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"thread_start_us":402,"threads_started":1}
I20260928 14:57:00.349814 6112 heartbeater.cc:347] Connected to a master server at 127.31.40.62:32909
I20260928 14:57:00.350188 6112 heartbeater.cc:464] Registering TS with master...
I20260928 14:57:00.351052 6112 heartbeater.cc:511] Master 127.31.40.62:32909 requested a full tablet report, sending...
I20260928 14:57:00.352756 5655 ts_manager.cc:194] Registered new tserver with Master: f2f0f28f0299472386abafe15704df7b (127.31.40.3:34255)
I20260928 14:57:00.353735 5655 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.40.3@KRBTEST.COM'} at 127.31.40.3:51283
I20260928 14:57:00.356082 31904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.31.40.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.40.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.40.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/krb5kdc/oryx_127.31.40.61.keytab.
Entry for principal oryx/127.31.40.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/krb5kdc/oryx_127.31.40.61.keytab.
I20260928 14:57:00.408496 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.61:37667
--webserver_interface=127.31.40.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/krb5kdc/oryx_127.31.40.61.keytab
--principal=oryx/127.31.40.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:32961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.40.62:32909,127.31.40.61:37667
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:57:00.750432 6117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:00.750727 6117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:00.750815 6117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:00.759300 6117 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:57:00.759452 6117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:00.759509 6117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:00.759564 6117 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:57:00.759605 6117 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:57:00.771503 6117 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:32961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.40.62:32909,127.31.40.61:37667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/krb5kdc/oryx_127.31.40.61.keytab
--principal=oryx/127.31.40.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.61:37667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.40.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
W20260928 14:57:00.771919 6117 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.31.40.62:32909,127.31.40.61:37667'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 14:57:00.773228 6117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:00.774710 6117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:00.787957 6125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:00.787896 6123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:00.788234 6122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:00.789239 6117 server_base.cc:1034] running on GCE node
I20260928 14:57:00.790110 6117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:00.791551 6117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:00.792826 6117 hybrid_clock.cc:648] HybridClock initialized: now 1790607420792736 us; error 65 us; skew 500 ppm
I20260928 14:57:00.793246 6117 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:57:00 dist-test-slave-j968 krb5kdc[5610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607420, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.40.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:00.796574 6117 init.cc:377] Logged in from keytab as oryx/127.31.40.61@KRBTEST.COM (short username oryx)
I20260928 14:57:00.798319 6117 webserver.cc:460] Webserver started at http://127.31.40.61:40559/ using document root <none> and password file <none>
I20260928 14:57:00.799121 6117 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:00.799268 6117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:00.799631 6117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:00.803045 6117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-1/data/instance:
uuid: "791fc9b073e5437e8af2e1379eb4fe02"
format_stamp: "Formatted at 2026-09-28 14:57:00 on dist-test-slave-j968"
server_key: "298c34f045005fbddbe66b82c858fc24"
server_key_iv: "4ffa7587d20f9116bfa809f07310f18b"
server_key_version: "encryptionkey@0"
I20260928 14:57:00.803922 6117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-1/wal/instance:
uuid: "791fc9b073e5437e8af2e1379eb4fe02"
format_stamp: "Formatted at 2026-09-28 14:57:00 on dist-test-slave-j968"
server_key: "298c34f045005fbddbe66b82c858fc24"
server_key_iv: "4ffa7587d20f9116bfa809f07310f18b"
server_key_version: "encryptionkey@0"
I20260928 14:57:00.809764 6117 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 14:57:00.813591 6132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:00.815172 6117 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 14:57:00.815392 6117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-1/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-1/wal
uuid: "791fc9b073e5437e8af2e1379eb4fe02"
format_stamp: "Formatted at 2026-09-28 14:57:00 on dist-test-slave-j968"
server_key: "298c34f045005fbddbe66b82c858fc24"
server_key_iv: "4ffa7587d20f9116bfa809f07310f18b"
server_key_version: "encryptionkey@0"
I20260928 14:57:00.815630 6117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:00.841089 5978 heartbeater.cc:503] Master 127.31.40.62:32909 was elected leader, sending a full tablet report...
I20260928 14:57:00.846524 6117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:00.851555 6117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:00.851924 6117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:00.872218 6117 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.61:37667
I20260928 14:57:00.872246 6183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.61:37667 every 8 connection(s)
I20260928 14:57:00.873703 6117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-1/data/info.pb
I20260928 14:57:00.877638 6184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:00.881291 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 6117
I20260928 14:57:00.885358 6184 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32909 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:57:00 dist-test-slave-j968 krb5kdc[5610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607420, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.40.61@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM
Sep 28 14:57:00 dist-test-slave-j968 krb5kdc[5610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.40.61@KRBTEST.COM
I20260928 14:57:00.905527 6188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:00.889410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59332 (local address 127.31.40.62:32909)
0928 14:57:00.889910 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:00.889918 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:00.890383 (+ 465us) server_negotiation.cc:407] Connection header received
0928 14:57:00.890605 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:00.890610 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:00.890681 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:00.890793 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:00.891924 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:00.892665 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:00.894053 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:00.894451 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:00.894582 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:00.898114 (+ 3532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:00.898119 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:00.898123 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:00.898156 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:00.901639 (+ 3483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:00.902605 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:00.902608 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:00.902611 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:00.902727 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:00.903248 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:00.903253 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:00.903257 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:00.903631 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:00.903770 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:00.904841 (+ 1071us) server_negotiation.cc:299] Negotiation successful
0928 14:57:00.905160 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":275,"threads_started":1}
I20260928 14:57:00.906576 6187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:00.883634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58286 (local address 127.31.40.61:37667)
0928 14:57:00.884514 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:00.884526 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:00.884569 (+ 43us) server_negotiation.cc:407] Connection header received
0928 14:57:00.886433 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:00.886459 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:00.886896 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:00.887295 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:00.889149 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:00.891186 (+ 2037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:00.892664 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:00.893418 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:00.893627 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:00.898667 (+ 5040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:00.898676 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:00.898690 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:00.898721 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:00.902302 (+ 3581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:00.903163 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:00.903170 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:00.903172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:00.903348 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:00.903805 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:00.903812 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:00.903819 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:00.904269 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:00.904535 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:00.904966 (+ 431us) server_negotiation.cc:299] Negotiation successful
0928 14:57:00.905276 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":429,"threads_started":1}
I20260928 14:57:00.906739 6189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:00.889425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:32909 (local address 127.0.0.1:59332)
0928 14:57:00.890249 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:00.890291 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:00.890463 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:00.890856 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:00.890863 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:00.891505 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:00.891755 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:00.891764 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:00.892804 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:00.892810 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:00.893909 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:00.893920 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:00.894680 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:00.894688 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:00.894834 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:00.895452 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:00.895474 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:00.897907 (+ 2433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:00.901792 (+ 3885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:00.901802 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:00.901813 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:00.902478 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:00.902842 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:00.902848 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:00.902855 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:00.903125 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:00.903778 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:00.903788 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:00.904356 (+ 568us) client_negotiation.cc:769] Sending connection context
0928 14:57:00.904739 (+ 383us) client_negotiation.cc:240] Negotiation successful
0928 14:57:00.905620 (+ 881us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"mutex_wait_us":168,"spinlock_wait_cycles":30464,"thread_start_us":439,"threads_started":1}
I20260928 14:57:00.909279 6184 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.40.61" port: 37667 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:57:00 dist-test-slave-j968 krb5kdc[5610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607420, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.40.61@KRBTEST.COM for oryx/127.31.40.61@KRBTEST.COM
I20260928 14:57:00.923243 6189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:00.910125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.61:37667 (local address 127.0.0.1:58302)
0928 14:57:00.910269 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:00.910283 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:00.910364 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:00.910709 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:00.910713 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:00.911022 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:00.911150 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:00.911158 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:00.912270 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:00.912274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:00.913195 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:00.913212 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:00.913923 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:00.913926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:00.914061 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:00.914594 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:00.914610 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:00.917150 (+ 2540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:00.920517 (+ 3367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:00.920525 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:00.920530 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:00.920990 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:00.921383 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:00.921391 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:00.921395 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:00.921553 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:00.922182 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:00.922187 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:00.922336 (+ 149us) client_negotiation.cc:769] Sending connection context
0928 14:57:00.922485 (+ 149us) client_negotiation.cc:240] Negotiation successful
0928 14:57:00.922793 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 14:57:00.923278 6187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:00.910320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58302 (local address 127.31.40.61:37667)
0928 14:57:00.910486 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:00.910491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:00.910505 (+ 14us) server_negotiation.cc:407] Connection header received
0928 14:57:00.910544 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:00.910547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:00.910599 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:00.910680 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:00.911266 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:00.912132 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:00.913329 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:00.913802 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:00.913903 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:00.917341 (+ 3438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:00.917347 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:00.917352 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:00.917392 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:00.920361 (+ 2969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:00.921124 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:00.921133 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:00.921137 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:00.921267 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:00.921675 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:00.921682 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:00.921685 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:00.922040 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:00.922210 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:00.922567 (+ 357us) server_negotiation.cc:299] Negotiation successful
0928 14:57:00.922944 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":161,"server-negotiator.queue_time_us":54}
I20260928 14:57:00.928635 6184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02: Bootstrap starting.
I20260928 14:57:00.932066 6184 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:00.933213 6184 log.cc:824] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:00.936268 6184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02: No bootstrap required, opened a new log
I20260928 14:57:00.944032 6184 raft_consensus.cc:348] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40f51af6fc714868a4c4df423f897b6c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32909 } } peers { permanent_uuid: "791fc9b073e5437e8af2e1379eb4fe02" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 37667 } }
I20260928 14:57:00.944446 6184 raft_consensus.cc:374] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:00.944567 6184 raft_consensus.cc:729] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 791fc9b073e5437e8af2e1379eb4fe02, State: Initialized, Role: FOLLOWER
I20260928 14:57:00.945328 6184 consensus_queue.cc:260] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40f51af6fc714868a4c4df423f897b6c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32909 } } peers { permanent_uuid: "791fc9b073e5437e8af2e1379eb4fe02" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 37667 } }
I20260928 14:57:00.946334 6191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40f51af6fc714868a4c4df423f897b6c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32909 } } peers { permanent_uuid: "791fc9b073e5437e8af2e1379eb4fe02" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 37667 } } }
I20260928 14:57:00.946650 6191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:57:00.947441 6184 sys_catalog.cc:564] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 14:57:00.956679 6202 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:57:00.956841 6202 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:57:00.959728 6117 master_runner.cc:428] Detected that this master 791fc9b073e5437e8af2e1379eb4fe02 is joining an existing cluster
I20260928 14:57:00.959882 6117 master_runner.cc:432] Initiating AddMaster RPC to add 127.31.40.61:37667
WARNING: no policy specified for oryx/127.31.40.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.40.60@KRBTEST.COM" created.
I20260928 14:57:00.995565 6187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:00.965530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58318 (local address 127.31.40.61:37667)
0928 14:57:00.965727 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:00.965735 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:00.976198 (+ 10463us) server_negotiation.cc:407] Connection header received
0928 14:57:00.976689 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:00.976695 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:00.976770 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:00.976858 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:00.978902 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:00.980225 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:00.982743 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:00.983586 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:00.983698 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:00.987485 (+ 3787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:00.987492 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:00.987498 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:00.987540 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:00.991747 (+ 4207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:00.992905 (+ 1158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:00.992911 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:00.992915 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:00.993099 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:00.993643 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:00.993648 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:00.993651 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:00.994060 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:00.994234 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:00.994871 (+ 637us) server_negotiation.cc:299] Negotiation successful
0928 14:57:00.995138 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 14:57:00.995554 6188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:00.965717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59334 (local address 127.31.40.62:32909)
0928 14:57:00.965926 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:00.965936 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:00.976035 (+ 10099us) server_negotiation.cc:407] Connection header received
0928 14:57:00.976672 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:00.976681 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:00.976782 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:00.976869 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:00.979016 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:00.980326 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:00.982908 (+ 2582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:00.983703 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:00.983830 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:00.987687 (+ 3857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:00.987696 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:00.987702 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:00.987750 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:00.991927 (+ 4177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:00.992959 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:00.992965 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:00.992969 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:00.993141 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:00.993668 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:00.993674 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:00.993677 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:00.994060 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:00.994215 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:00.994891 (+ 676us) server_negotiation.cc:299] Negotiation successful
0928 14:57:00.995143 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 14:57:00.996755 6209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:00.964934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:32909 (local address 127.0.0.1:59334)
0928 14:57:00.975848 (+ 10914us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:00.975914 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:00.976492 (+ 578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:00.977111 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:00.977118 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:00.978152 (+ 1034us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:00.978771 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:00.978787 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:00.980510 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:00.980517 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:00.982717 (+ 2200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:00.982733 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:00.984209 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:00.984217 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:00.984608 (+ 391us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:00.985770 (+ 1162us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:00.985804 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:00.987402 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:00.992128 (+ 4726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:00.992134 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:00.992140 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:00.992854 (+ 714us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:00.993285 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:00.993290 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:00.993295 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:00.993508 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:00.994294 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:00.994299 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:00.994560 (+ 261us) client_negotiation.cc:769] Sending connection context
0928 14:57:00.995896 (+ 1336us) client_negotiation.cc:240] Negotiation successful
0928 14:57:00.996223 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10699,"mutex_wait_us":187,"spinlock_wait_cycles":5760,"thread_start_us":11253,"threads_started":1}
I20260928 14:57:00.997558 6208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:00.965024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.61:37667 (local address 127.0.0.1:58318)
0928 14:57:00.976080 (+ 11056us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:00.976107 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:00.976544 (+ 437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:00.977152 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:00.977158 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:00.978156 (+ 998us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:00.978712 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:00.978726 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:00.980492 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:00.980498 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:00.982566 (+ 2068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:00.982580 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:00.984181 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:00.984189 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:00.984479 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:00.985634 (+ 1155us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:00.985684 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:00.987278 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:00.992022 (+ 4744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:00.992031 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:00.992040 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:00.992748 (+ 708us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:00.993262 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:00.993271 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:00.993277 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:00.993453 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:00.994417 (+ 964us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:00.994425 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:00.994560 (+ 135us) client_negotiation.cc:769] Sending connection context
0928 14:57:00.996300 (+ 1740us) client_negotiation.cc:240] Negotiation successful
0928 14:57:00.996623 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10895,"mutex_wait_us":169,"spinlock_wait_cycles":34688,"thread_start_us":11030,"threads_started":1}
Sep 28 14:57:01 dist-test-slave-j968 krb5kdc[5610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607418, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.40.62@KRBTEST.COM for oryx/127.31.40.61@KRBTEST.COM
I20260928 14:57:01.024223 6187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:01.004121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58320 (local address 127.31.40.61:37667)
0928 14:57:01.004394 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:01.004403 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:01.008659 (+ 4256us) server_negotiation.cc:407] Connection header received
0928 14:57:01.009069 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:01.009073 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:01.009149 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:01.009274 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:01.010527 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:01.011384 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:01.012913 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:01.013436 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:01.013604 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:01.017355 (+ 3751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:01.017361 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:01.017365 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:01.017399 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:01.021050 (+ 3651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:01.021843 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:01.021849 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:01.021866 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:01.022018 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:01.022470 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:01.022475 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:01.022480 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:01.022788 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:01.022960 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:01.023501 (+ 541us) server_negotiation.cc:299] Negotiation successful
0928 14:57:01.023752 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260928 14:57:01.024276 6212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:01.003838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.61:37667 (local address 127.0.0.1:58320)
0928 14:57:01.008514 (+ 4676us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:01.008560 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:01.008809 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:01.009330 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:01.009338 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:01.010102 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:01.010342 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:01.010353 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:01.011534 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:01.011542 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:01.012742 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:01.012756 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:01.013629 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:01.013636 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:01.013862 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:01.014608 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:01.014629 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:01.017133 (+ 2504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:01.021207 (+ 4074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:01.021223 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:01.021232 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:01.021699 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:01.022142 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:01.022148 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:01.022152 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:01.022338 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:01.022982 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:01.022993 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:01.023186 (+ 193us) client_negotiation.cc:769] Sending connection context
0928 14:57:01.023392 (+ 206us) client_negotiation.cc:240] Negotiation successful
0928 14:57:01.023778 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4405,"thread_start_us":4217,"threads_started":1}
I20260928 14:57:01.026939 5655 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.31.40.61:37667: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "791fc9b073e5437e8af2e1379eb4fe02"
member_type: NON_VOTER
last_known_addr {
host: "127.31.40.61"
port: 37667
}
attrs {
promote: true
}
}
dest_uuid: "40f51af6fc714868a4c4df423f897b6c"
cas_config_opid_index: -1
I20260928 14:57:01.028061 5655 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40f51af6fc714868a4c4df423f897b6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "40f51af6fc714868a4c4df423f897b6c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32909 } } peers { permanent_uuid: "791fc9b073e5437e8af2e1379eb4fe02" member_type: NON_VOTER last_known_addr { host: "127.31.40.61" port: 37667 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.40.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/krb5kdc/oryx_127.31.40.60.keytab.
Entry for principal oryx/127.31.40.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/krb5kdc/oryx_127.31.40.60.keytab.
I20260928 14:57:01.031445 6215 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 40f51af6fc714868a4c4df423f897b6c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 791fc9b073e5437e8af2e1379eb4fe02 (127.31.40.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "40f51af6fc714868a4c4df423f897b6c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32909 } } peers { permanent_uuid: "791fc9b073e5437e8af2e1379eb4fe02" member_type: NON_VOTER last_known_addr { host: "127.31.40.61" port: 37667 } attrs { promote: true } } }
I20260928 14:57:01.032588 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.60:42157
--webserver_interface=127.31.40.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/krb5kdc/oryx_127.31.40.60.keytab
--principal=oryx/127.31.40.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:32961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.40.62:32909,127.31.40.61:37667,127.31.40.60:42157
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:57:01.034991 6215 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.31.40.61:37667
I20260928 14:57:01.035903 6117 master.cc:537] Master@127.31.40.61:37667 shutting down...
I20260928 14:57:01.037065 6216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40f51af6fc714868a4c4df423f897b6c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "40f51af6fc714868a4c4df423f897b6c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "40f51af6fc714868a4c4df423f897b6c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32909 } } peers { permanent_uuid: "791fc9b073e5437e8af2e1379eb4fe02" member_type: NON_VOTER last_known_addr { host: "127.31.40.61" port: 37667 } attrs { promote: true } } }
I20260928 14:57:01.037439 6216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40f51af6fc714868a4c4df423f897b6c [sys.catalog]: This master's current role is: LEADER
W20260928 14:57:01.042188 5641 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260928 14:57:01.045218 5641 consensus_peers.cc:487] T 00000000000000000000000000000000 P 40f51af6fc714868a4c4df423f897b6c -> Peer 791fc9b073e5437e8af2e1379eb4fe02 (127.31.40.61:37667): Couldn't send request to peer 791fc9b073e5437e8af2e1379eb4fe02. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 14:57:01.050982 6117 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260928 14:57:01.051465 6117 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260928 14:57:01.051681 6117 tablet_replica.cc:330] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02: stopping tablet replica
I20260928 14:57:01.059283 6117 master.cc:552] Master@127.31.40.61:37667 shutdown complete.
I20260928 14:57:01.059386 6117 master_runner.cc:305] Clearing existing system tablet
I20260928 14:57:01.059948 6117 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:01.060077 6117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:01.064580 6117 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260928 14:57:01.068233 6222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:01.068634 6117 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:01.069006 6117 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:57:01.069108 6117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-1/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-1/wal
uuid: "791fc9b073e5437e8af2e1379eb4fe02"
format_stamp: "Formatted at 2026-09-28 14:57:00 on dist-test-slave-j968"
server_key: "298c34f045005fbddbe66b82c858fc24"
server_key_iv: "4ffa7587d20f9116bfa809f07310f18b"
server_key_version: "encryptionkey@0"
I20260928 14:57:01.074739 6117 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:57:01.082780 6117 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 14:57:01.083017 6117 log.cc:1196] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 14:57:01.083621 6117 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02: Deleting consensus metadata
I20260928 14:57:01.083943 6117 master_runner.cc:315] Copying system tablet from 127.31.40.62:32909
I20260928 14:57:01.089334 6117 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02: tablet copy: Beginning tablet copy session from remote peer at address 127.31.40.62:32909
I20260928 14:57:01.106578 6188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:01.090251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59350 (local address 127.31.40.62:32909)
0928 14:57:01.090459 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:01.090467 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:01.091045 (+ 578us) server_negotiation.cc:407] Connection header received
0928 14:57:01.091182 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:01.091187 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:01.091321 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:01.091439 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:01.092630 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:01.093972 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:01.096134 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:01.096864 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:01.097016 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:01.099500 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:01.099506 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:01.099511 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:01.099551 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:01.103313 (+ 3762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:01.104217 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:01.104223 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:01.104226 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:01.104381 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:01.104838 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:01.104843 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:01.104845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:01.105276 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:01.105429 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:01.105962 (+ 533us) server_negotiation.cc:299] Negotiation successful
0928 14:57:01.106190 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 14:57:01.106804 6228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:01.090191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:32909 (local address 127.0.0.1:59350)
0928 14:57:01.090839 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:01.090866 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:01.091034 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:01.091501 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:01.091507 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:01.092131 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:01.092441 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:01.092454 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:01.094144 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:01.094156 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:01.095953 (+ 1797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:01.095972 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:01.097056 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:01.097063 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:01.097266 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:01.098035 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:01.098060 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:01.099244 (+ 1184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:01.103480 (+ 4236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:01.103487 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:01.103492 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:01.104076 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:01.104510 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:01.104516 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:01.104520 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:01.104703 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:01.105421 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:01.105426 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:01.105628 (+ 202us) client_negotiation.cc:769] Sending connection context
0928 14:57:01.105804 (+ 176us) client_negotiation.cc:240] Negotiation successful
0928 14:57:01.106171 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":393,"threads_started":1}
I20260928 14:57:01.108019 5675 tablet_copy_service.cc:140] P 40f51af6fc714868a4c4df423f897b6c: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 791fc9b073e5437e8af2e1379eb4fe02 ({username='oryx', principal='oryx/127.31.40.61@KRBTEST.COM'} at 127.0.0.1:59350)
I20260928 14:57:01.108330 5675 tablet_copy_service.cc:161] P 40f51af6fc714868a4c4df423f897b6c: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 791fc9b073e5437e8af2e1379eb4fe02 at {username='oryx', principal='oryx/127.31.40.61@KRBTEST.COM'} at 127.0.0.1:59350: session id = 791fc9b073e5437e8af2e1379eb4fe02-00000000000000000000000000000000
I20260928 14:57:01.112751 5675 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 40f51af6fc714868a4c4df423f897b6c: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 14:57:01.115561 6117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 14:57:01.123440 6117 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02: tablet copy: Starting download of 0 data blocks...
I20260928 14:57:01.123808 6117 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02: tablet copy: Starting download of 1 WAL segments...
I20260928 14:57:01.127058 6117 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 14:57:01.132405 5675 tablet_copy_service.cc:342] P 40f51af6fc714868a4c4df423f897b6c: Request end of tablet copy session 791fc9b073e5437e8af2e1379eb4fe02-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.31.40.61@KRBTEST.COM'} at 127.0.0.1:59350
I20260928 14:57:01.132660 5675 tablet_copy_service.cc:433] P 40f51af6fc714868a4c4df423f897b6c: ending tablet copy session 791fc9b073e5437e8af2e1379eb4fe02-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 791fc9b073e5437e8af2e1379eb4fe02
I20260928 14:57:01.145581 6117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:01.148928 6229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:01.159752 6232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:01.160315 6230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:01.162866 6117 server_base.cc:1034] running on GCE node
I20260928 14:57:01.163354 6117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:01.164228 6117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:01.165607 6117 hybrid_clock.cc:648] HybridClock initialized: now 1790607421165534 us; error 43 us; skew 500 ppm
I20260928 14:57:01.165879 6117 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:57:01 dist-test-slave-j968 krb5kdc[5610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607421, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.40.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:01.168380 6117 init.cc:377] Logged in from keytab as oryx/127.31.40.61@KRBTEST.COM (short username oryx)
I20260928 14:57:01.170192 6117 webserver.cc:460] Webserver started at http://127.31.40.61:40489/ using document root <none> and password file <none>
I20260928 14:57:01.170606 6117 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:01.170694 6117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:01.176788 6117 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 14:57:01.179111 6239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:01.179697 6117 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:57:01.179942 6117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-1/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-1/wal
uuid: "791fc9b073e5437e8af2e1379eb4fe02"
format_stamp: "Formatted at 2026-09-28 14:57:00 on dist-test-slave-j968"
server_key: "298c34f045005fbddbe66b82c858fc24"
server_key_iv: "4ffa7587d20f9116bfa809f07310f18b"
server_key_version: "encryptionkey@0"
I20260928 14:57:01.180212 6117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:01.201934 6117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:01.206518 6117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:01.233440 6117 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.61:37667
I20260928 14:57:01.233569 6290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.61:37667 every 8 connection(s)
I20260928 14:57:01.234753 6117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-1/data/info.pb
I20260928 14:57:01.240981 6291 sys_catalog.cc:263] Verifying existing consensus state
I20260928 14:57:01.245502 6291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02: Bootstrap starting.
I20260928 14:57:01.264137 6291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:57:01.264694 6291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02: Bootstrap complete.
I20260928 14:57:01.265795 6291 raft_consensus.cc:348] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "40f51af6fc714868a4c4df423f897b6c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32909 } } peers { permanent_uuid: "791fc9b073e5437e8af2e1379eb4fe02" member_type: NON_VOTER last_known_addr { host: "127.31.40.61" port: 37667 } attrs { promote: true } }
I20260928 14:57:01.266070 6291 raft_consensus.cc:729] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 791fc9b073e5437e8af2e1379eb4fe02, State: Initialized, Role: LEARNER
I20260928 14:57:01.266314 6291 consensus_queue.cc:260] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "40f51af6fc714868a4c4df423f897b6c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32909 } } peers { permanent_uuid: "791fc9b073e5437e8af2e1379eb4fe02" member_type: NON_VOTER last_known_addr { host: "127.31.40.61" port: 37667 } attrs { promote: true } }
I20260928 14:57:01.267418 6291 sys_catalog.cc:564] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:57:01.267174 6294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "40f51af6fc714868a4c4df423f897b6c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32909 } } peers { permanent_uuid: "791fc9b073e5437e8af2e1379eb4fe02" member_type: NON_VOTER last_known_addr { host: "127.31.40.61" port: 37667 } attrs { promote: true } } }
I20260928 14:57:01.267691 6294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [sys.catalog]: This master's current role is: LEARNER
I20260928 14:57:01.280668 6305 catalog_manager.cc:1259] Loaded cluster ID: 913af42657514f2aa51a685dba5d7ba1
I20260928 14:57:01.280869 6305 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02: loading cluster ID for follower catalog manager: success
I20260928 14:57:01.284538 6305 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02: acquiring CA information for follower catalog manager: success
I20260928 14:57:01.287294 6305 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 14:57:01.357056 6112 heartbeater.cc:503] Master 127.31.40.62:32909 was elected leader, sending a full tablet report...
W20260928 14:57:01.434366 6219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:01.434712 6219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:01.434875 6219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:01.442653 6219 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:57:01.442829 6219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:01.442945 6219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:01.443061 6219 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:57:01.443140 6219 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:57:01.454144 6219 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:32961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.40.62:32909,127.31.40.61:37667,127.31.40.60:42157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/krb5kdc/oryx_127.31.40.60.keytab
--principal=oryx/127.31.40.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.60:42157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.40.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:57:01.455600 6219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:01.457084 6219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:01.471477 6310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:01.471330 6311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:01.472203 6219 server_base.cc:1034] running on GCE node
W20260928 14:57:01.472360 6313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:01.473177 6219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:01.474771 6219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:01.476033 6219 hybrid_clock.cc:648] HybridClock initialized: now 1790607421475918 us; error 48 us; skew 500 ppm
I20260928 14:57:01.476388 6219 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:57:01 dist-test-slave-j968 krb5kdc[5610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607421, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.40.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:01.479638 6219 init.cc:377] Logged in from keytab as oryx/127.31.40.60@KRBTEST.COM (short username oryx)
I20260928 14:57:01.481119 6219 webserver.cc:460] Webserver started at http://127.31.40.60:40611/ using document root <none> and password file <none>
I20260928 14:57:01.481778 6219 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:01.481886 6219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:01.482234 6219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:01.485493 6219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-2/data/instance:
uuid: "156238ac7d104bae8996c9dd47abd3cb"
format_stamp: "Formatted at 2026-09-28 14:57:01 on dist-test-slave-j968"
server_key: "f26f1d5ae3f5abc4264aba9561b977e4"
server_key_iv: "6b319ffd65979c5f6f9a2775e6988c01"
server_key_version: "encryptionkey@0"
I20260928 14:57:01.486438 6219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-2/wal/instance:
uuid: "156238ac7d104bae8996c9dd47abd3cb"
format_stamp: "Formatted at 2026-09-28 14:57:01 on dist-test-slave-j968"
server_key: "f26f1d5ae3f5abc4264aba9561b977e4"
server_key_iv: "6b319ffd65979c5f6f9a2775e6988c01"
server_key_version: "encryptionkey@0"
I20260928 14:57:01.491717 6219 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 14:57:01.495477 6320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:01.497009 6219 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:57:01.497149 6219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-2/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-2/wal
uuid: "156238ac7d104bae8996c9dd47abd3cb"
format_stamp: "Formatted at 2026-09-28 14:57:01 on dist-test-slave-j968"
server_key: "f26f1d5ae3f5abc4264aba9561b977e4"
server_key_iv: "6b319ffd65979c5f6f9a2775e6988c01"
server_key_version: "encryptionkey@0"
I20260928 14:57:01.497313 6219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:01.517653 6219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:01.522199 6219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:01.522440 6219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:01.551923 6219 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.60:42157
I20260928 14:57:01.551951 6371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.60:42157 every 8 connection(s)
I20260928 14:57:01.553570 6219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-2/data/info.pb
I20260928 14:57:01.556156 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 6219
I20260928 14:57:01.558534 6372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:01.568570 6372 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32909 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:57:01 dist-test-slave-j968 krb5kdc[5610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.40.60@KRBTEST.COM
Sep 28 14:57:01 dist-test-slave-j968 krb5kdc[5610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607421, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.40.60@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM
I20260928 14:57:01.585273 6375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:01.558502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45152 (local address 127.31.40.60:42157)
0928 14:57:01.559576 (+ 1074us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:01.559588 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:01.559630 (+ 42us) server_negotiation.cc:407] Connection header received
0928 14:57:01.561296 (+ 1666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:01.561319 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:01.561724 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:01.562105 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:01.564161 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:01.566020 (+ 1859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:01.568221 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:01.569331 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:01.569501 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:01.575369 (+ 5868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:01.575377 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:01.575388 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:01.575447 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:01.578988 (+ 3541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:01.580043 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:01.580050 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:01.580057 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:01.580253 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:01.580824 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:01.580830 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:01.580834 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:01.581680 (+ 846us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:01.581968 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:01.582613 (+ 645us) server_negotiation.cc:299] Negotiation successful
0928 14:57:01.583221 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":814,"thread_start_us":506,"threads_started":1}
I20260928 14:57:01.589776 6188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:01.572061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59366 (local address 127.31.40.62:32909)
0928 14:57:01.572317 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:01.572327 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:01.573101 (+ 774us) server_negotiation.cc:407] Connection header received
0928 14:57:01.573432 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:01.573436 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:01.573584 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:01.573716 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:01.574953 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:01.575910 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:01.577666 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:01.578054 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:01.578192 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:01.582451 (+ 4259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:01.582459 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:01.582465 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:01.582523 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:01.586523 (+ 4000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:01.587529 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:01.587536 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:01.587540 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:01.587713 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:01.588168 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:01.588177 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:01.588182 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:01.588483 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:01.588666 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:01.589145 (+ 479us) server_negotiation.cc:299] Negotiation successful
0928 14:57:01.589388 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 14:57:01.589884 6376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:01.572038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:32909 (local address 127.0.0.1:59366)
0928 14:57:01.572920 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:01.572988 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:01.573292 (+ 304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:01.573776 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:01.573781 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:01.574528 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:01.574784 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:01.574793 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:01.576119 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:01.576126 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:01.577465 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:01.577480 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:01.578253 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:01.578261 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:01.578438 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:01.579104 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:01.579132 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:01.582249 (+ 3117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:01.586728 (+ 4479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:01.586738 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:01.586745 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:01.587361 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:01.587850 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:01.587856 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:01.587860 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:01.588040 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:01.588624 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:01.588632 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:01.588826 (+ 194us) client_negotiation.cc:769] Sending connection context
0928 14:57:01.589025 (+ 199us) client_negotiation.cc:240] Negotiation successful
0928 14:57:01.589388 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":391,"threads_started":1}
I20260928 14:57:01.591251 6372 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.40.61" port: 37667 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:57:01 dist-test-slave-j968 krb5kdc[5610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607421, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.40.60@KRBTEST.COM for oryx/127.31.40.61@KRBTEST.COM
I20260928 14:57:01.605638 6376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:01.592305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.61:37667 (local address 127.0.0.1:58328)
0928 14:57:01.592565 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:01.592588 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:01.592700 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:01.593039 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:01.593043 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:01.593397 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:01.593577 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:01.593587 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:01.594967 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:01.594970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:01.596072 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:01.596086 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:01.596753 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:01.596758 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:01.596894 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:01.597463 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:01.597482 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:01.600105 (+ 2623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:01.603254 (+ 3149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:01.603258 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:01.603260 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:01.603659 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:01.604178 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:01.604182 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:01.604185 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:01.604339 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:01.604833 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:01.604837 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:01.604970 (+ 133us) client_negotiation.cc:769] Sending connection context
0928 14:57:01.605111 (+ 141us) client_negotiation.cc:240] Negotiation successful
0928 14:57:01.605361 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 14:57:01.605669 6377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:01.592223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58328 (local address 127.31.40.61:37667)
0928 14:57:01.592777 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:01.592782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:01.592798 (+ 16us) server_negotiation.cc:407] Connection header received
0928 14:57:01.592846 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:01.592850 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:01.592910 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:01.593002 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:01.593721 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:01.594840 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:01.596200 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:01.596608 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:01.596700 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:01.600315 (+ 3615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:01.600319 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:01.600323 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:01.600355 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:01.603118 (+ 2763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:01.603809 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:01.603814 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:01.603816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:01.604046 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:01.604458 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:01.604462 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:01.604464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:01.604731 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:01.604850 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:01.605220 (+ 370us) server_negotiation.cc:299] Negotiation successful
0928 14:57:01.605393 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":304,"threads_started":1}
I20260928 14:57:01.607088 6372 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.40.60" port: 42157 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:57:01 dist-test-slave-j968 krb5kdc[5610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607421, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.40.60@KRBTEST.COM for oryx/127.31.40.60@KRBTEST.COM
I20260928 14:57:01.621138 6376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:01.607740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:42157 (local address 127.0.0.1:45154)
0928 14:57:01.608030 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:01.608074 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:01.608184 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:01.608472 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:01.608477 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:01.608913 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:01.609063 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:01.609072 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:01.610103 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:01.610112 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:01.611053 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:01.611065 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:01.611794 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:01.611800 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:01.612048 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:01.612686 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:01.612705 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:01.614940 (+ 2235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:01.618480 (+ 3540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:01.618497 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:01.618502 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:01.618868 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:01.619261 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:01.619268 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:01.619272 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:01.619425 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:01.620008 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:01.620013 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:01.620195 (+ 182us) client_negotiation.cc:769] Sending connection context
0928 14:57:01.620358 (+ 163us) client_negotiation.cc:240] Negotiation successful
0928 14:57:01.620673 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 14:57:01.621241 6375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:01.608019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45154 (local address 127.31.40.60:42157)
0928 14:57:01.608181 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:01.608187 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:01.608204 (+ 17us) server_negotiation.cc:407] Connection header received
0928 14:57:01.608288 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:01.608294 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:01.608356 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:01.608457 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:01.609200 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:01.609965 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:01.611195 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:01.611632 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:01.611779 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:01.615199 (+ 3420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:01.615205 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:01.615209 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:01.615237 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:01.618335 (+ 3098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:01.619016 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:01.619020 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:01.619023 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:01.619147 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:01.619545 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:01.619553 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:01.619557 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:01.619836 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:01.620120 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:01.620448 (+ 328us) server_negotiation.cc:299] Negotiation successful
0928 14:57:01.620839 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":160,"server-negotiator.queue_time_us":56}
I20260928 14:57:01.626596 6372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb: Bootstrap starting.
I20260928 14:57:01.630466 6372 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:01.631809 6372 log.cc:824] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:01.637537 6372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb: No bootstrap required, opened a new log
I20260928 14:57:01.649539 6377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:01.641710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58332 (local address 127.31.40.61:37667)
0928 14:57:01.641920 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:01.641928 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:01.642027 (+ 99us) server_negotiation.cc:407] Connection header received
0928 14:57:01.642215 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:01.642221 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:01.642331 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:01.642454 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:01.642941 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:01.644177 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:01.647028 (+ 2851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:01.648466 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:01.648605 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:01.648781 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:01.649121 (+ 340us) server_negotiation.cc:299] Negotiation successful
0928 14:57:01.649239 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 14:57:01.649595 6380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:01.641309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.61:37667 (local address 127.0.0.1:58332)
0928 14:57:01.641889 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:01.641911 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:01.642067 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:01.642524 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:01.642529 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:01.642557 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:01.642777 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:01.642788 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:01.644350 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:01.644356 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:01.646794 (+ 2438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:01.646815 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:01.648693 (+ 1878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:01.648701 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:01.648898 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:01.648972 (+ 74us) client_negotiation.cc:769] Sending connection context
0928 14:57:01.649086 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 14:57:01.649259 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":264,"threads_started":1}
I20260928 14:57:01.650772 6266 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20260928 14:57:01.650682 6372 raft_consensus.cc:348] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40f51af6fc714868a4c4df423f897b6c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32909 } } peers { permanent_uuid: "791fc9b073e5437e8af2e1379eb4fe02" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 37667 } } peers { permanent_uuid: "156238ac7d104bae8996c9dd47abd3cb" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 42157 } }
I20260928 14:57:01.651108 6372 raft_consensus.cc:374] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:01.651238 6372 raft_consensus.cc:729] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 156238ac7d104bae8996c9dd47abd3cb, State: Initialized, Role: FOLLOWER
I20260928 14:57:01.651531 6294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40f51af6fc714868a4c4df423f897b6c. Latest consensus state: current_term: 1 leader_uuid: "40f51af6fc714868a4c4df423f897b6c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "40f51af6fc714868a4c4df423f897b6c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32909 } } peers { permanent_uuid: "791fc9b073e5437e8af2e1379eb4fe02" member_type: NON_VOTER last_known_addr { host: "127.31.40.61" port: 37667 } attrs { promote: true } } }
I20260928 14:57:01.651789 6294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [sys.catalog]: This master's current role is: LEARNER
I20260928 14:57:01.651947 6372 consensus_queue.cc:260] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40f51af6fc714868a4c4df423f897b6c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32909 } } peers { permanent_uuid: "791fc9b073e5437e8af2e1379eb4fe02" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 37667 } } peers { permanent_uuid: "156238ac7d104bae8996c9dd47abd3cb" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 42157 } }
I20260928 14:57:01.652962 6381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40f51af6fc714868a4c4df423f897b6c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32909 } } peers { permanent_uuid: "791fc9b073e5437e8af2e1379eb4fe02" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 37667 } } peers { permanent_uuid: "156238ac7d104bae8996c9dd47abd3cb" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 42157 } } }
I20260928 14:57:01.653424 6381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:57:01.654085 6372 sys_catalog.cc:564] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb [sys.catalog]: configured and running, proceeding with master startup.
W20260928 14:57:01.665836 6392 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:57:01.666066 6392 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:57:01.670976 6219 master_runner.cc:428] Detected that this master 156238ac7d104bae8996c9dd47abd3cb is joining an existing cluster
I20260928 14:57:01.671129 6219 master_runner.cc:432] Initiating AddMaster RPC to add 127.31.40.60:42157
I20260928 14:57:01.692458 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 5716
I20260928 14:57:01.697726 6399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:01.677321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.61:37667 (local address 127.0.0.1:58344)
0928 14:57:01.679263 (+ 1942us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:01.679285 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:01.679598 (+ 313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:01.680500 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:01.680508 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:01.681818 (+ 1310us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:01.682333 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:01.682356 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:01.683612 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:01.683623 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:01.685903 (+ 2280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:01.685920 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:01.687154 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:01.687161 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:01.687401 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:01.688239 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:01.688264 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:01.689509 (+ 1245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:01.694545 (+ 5036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:01.694556 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:01.694562 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:01.695073 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:01.695522 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:01.695527 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:01.695531 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:01.695688 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:01.696434 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:01.696440 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:01.696655 (+ 215us) client_negotiation.cc:769] Sending connection context
0928 14:57:01.696807 (+ 152us) client_negotiation.cc:240] Negotiation successful
0928 14:57:01.697206 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1808,"mutex_wait_us":29,"thread_start_us":1573,"threads_started":1}
I20260928 14:57:01.698689 6377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:01.676958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58344 (local address 127.31.40.61:37667)
0928 14:57:01.677320 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:01.677327 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:01.679378 (+ 2051us) server_negotiation.cc:407] Connection header received
0928 14:57:01.679825 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:01.679834 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:01.679988 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:01.680092 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:01.682531 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:01.683425 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:01.686105 (+ 2680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:01.686773 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:01.686899 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:01.689746 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:01.689763 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:01.689768 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:01.689819 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:01.694369 (+ 4550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:01.695222 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:01.695229 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:01.695234 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:01.695386 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:01.695824 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:01.695829 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:01.695833 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:01.696228 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:01.696367 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:01.698113 (+ 1746us) server_negotiation.cc:299] Negotiation successful
0928 14:57:01.698345 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191}
I20260928 14:57:01.700706 6188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:01.677505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59382 (local address 127.31.40.62:32909)
0928 14:57:01.677689 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:01.677696 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:01.678357 (+ 661us) server_negotiation.cc:407] Connection header received
0928 14:57:01.678824 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:01.678834 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:01.678965 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:01.679089 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:01.683474 (+ 4385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:01.684901 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:01.687329 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:01.688106 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:01.688221 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:01.691454 (+ 3233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:01.691462 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:01.691478 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:01.691521 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:01.696318 (+ 4797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:01.697513 (+ 1195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:01.697523 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:01.697530 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:01.697732 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:01.698297 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:01.698306 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:01.698313 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:01.698787 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:01.698941 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:01.699815 (+ 874us) server_negotiation.cc:299] Negotiation successful
0928 14:57:01.700171 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 14:57:01.702261 6398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:01.677086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:32909 (local address 127.0.0.1:59382)
0928 14:57:01.678214 (+ 1128us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:01.678236 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:01.678602 (+ 366us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:01.679339 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:01.679344 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:01.682945 (+ 3601us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:01.683288 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:01.683300 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:01.685126 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:01.685133 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:01.687145 (+ 2012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:01.687161 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:01.688573 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:01.688581 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:01.688887 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:01.689668 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:01.689691 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:01.691143 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:01.696532 (+ 5389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:01.696541 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:01.696548 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:01.697314 (+ 766us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:01.697923 (+ 609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:01.697930 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:01.697937 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:01.698139 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:01.699036 (+ 897us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:01.699041 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:01.699335 (+ 294us) client_negotiation.cc:769] Sending connection context
0928 14:57:01.701056 (+ 1721us) client_negotiation.cc:240] Negotiation successful
0928 14:57:01.701453 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":976,"mutex_wait_us":48,"thread_start_us":776,"threads_started":1}
I20260928 14:57:01.712481 6375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:01.680235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45164 (local address 127.31.40.60:42157)
0928 14:57:01.680465 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:01.680473 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:01.682140 (+ 1667us) server_negotiation.cc:407] Connection header received
0928 14:57:01.682369 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:01.682375 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:01.682488 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:01.682589 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:01.685581 (+ 2992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:01.686981 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:01.689515 (+ 2534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:01.690411 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:01.690533 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:01.704071 (+ 13538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:01.704077 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:01.704082 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:01.704117 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:01.708726 (+ 4609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:01.709587 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:01.709593 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:01.709597 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:01.709752 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:01.710207 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:01.710214 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:01.710220 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:01.710785 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:01.711000 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:01.711683 (+ 683us) server_negotiation.cc:299] Negotiation successful
0928 14:57:01.711984 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":79}
I20260928 14:57:01.713801 6400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:01.679357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:42157 (local address 127.0.0.1:45164)
0928 14:57:01.682028 (+ 2671us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:01.682044 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:01.682221 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:01.683654 (+ 1433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:01.683659 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:01.684777 (+ 1118us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:01.685390 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:01.685410 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:01.687273 (+ 1863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:01.687282 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:01.689334 (+ 2052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:01.689353 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:01.701203 (+ 11850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:01.701210 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:01.701689 (+ 479us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:01.702617 (+ 928us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:01.702646 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:01.703840 (+ 1194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:01.708916 (+ 5076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:01.708921 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:01.708925 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:01.709458 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:01.709907 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:01.709915 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:01.709920 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:01.710075 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:01.711076 (+ 1001us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:01.711083 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:01.711300 (+ 217us) client_negotiation.cc:769] Sending connection context
0928 14:57:01.712749 (+ 1449us) client_negotiation.cc:240] Negotiation successful
0928 14:57:01.713131 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2477,"mutex_wait_us":33,"thread_start_us":2295,"threads_started":1}
Sep 28 14:57:01 dist-test-slave-j968 krb5kdc[5610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607418, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.40.62@KRBTEST.COM for oryx/127.31.40.60@KRBTEST.COM
Sep 28 14:57:01 dist-test-slave-j968 krb5kdc[5610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.40.1@KRBTEST.COM
I20260928 14:57:01.731225 6380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:01.709449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:42157 (local address 127.0.0.1:45168)
0928 14:57:01.709780 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:01.709799 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:01.709904 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:01.711377 (+ 1473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:01.711382 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:01.711853 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:01.712100 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:01.712113 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:01.714583 (+ 2470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:01.714588 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:01.716081 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:01.716099 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:01.717218 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:01.717226 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:01.717432 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:01.718338 (+ 906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:01.718366 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:01.722289 (+ 3923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:01.727630 (+ 5341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:01.727637 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:01.727642 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:01.728692 (+ 1050us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:01.729254 (+ 562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:01.729259 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:01.729263 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:01.729436 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:01.730172 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:01.730177 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:01.730363 (+ 186us) client_negotiation.cc:769] Sending connection context
0928 14:57:01.730518 (+ 155us) client_negotiation.cc:240] Negotiation successful
0928 14:57:01.730828 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212}
I20260928 14:57:01.731372 6402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:01.710022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45168 (local address 127.31.40.60:42157)
0928 14:57:01.710968 (+ 946us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:01.710976 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:01.711003 (+ 27us) server_negotiation.cc:407] Connection header received
0928 14:57:01.711094 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:01.711100 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:01.711183 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:01.712250 (+ 1067us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:01.712627 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:01.714432 (+ 1805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:01.716240 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:01.716954 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:01.717119 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:01.722679 (+ 5560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:01.722703 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:01.722710 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:01.722754 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:01.727424 (+ 4670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:01.728910 (+ 1486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:01.728917 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:01.728920 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:01.729100 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:01.729592 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:01.729597 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:01.729600 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:01.730013 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:01.730171 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:01.730675 (+ 504us) server_negotiation.cc:299] Negotiation successful
0928 14:57:01.730947 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":797,"thread_start_us":596,"threads_started":1}
I20260928 14:57:01.733407 5655 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.31.40.60:42157: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "156238ac7d104bae8996c9dd47abd3cb"
member_type: NON_VOTER
last_known_addr {
host: "127.31.40.60"
port: 42157
}
attrs {
promote: true
}
}
dest_uuid: "40f51af6fc714868a4c4df423f897b6c"
cas_config_opid_index: 5
I20260928 14:57:01.734534 5655 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40f51af6fc714868a4c4df423f897b6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "40f51af6fc714868a4c4df423f897b6c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32909 } } peers { permanent_uuid: "791fc9b073e5437e8af2e1379eb4fe02" member_type: NON_VOTER last_known_addr { host: "127.31.40.61" port: 37667 } attrs { promote: true } } peers { permanent_uuid: "156238ac7d104bae8996c9dd47abd3cb" member_type: NON_VOTER last_known_addr { host: "127.31.40.60" port: 42157 } attrs { promote: true } }
I20260928 14:57:01.734761 6401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:01.694392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46370 (local address 127.31.40.1:35061)
0928 14:57:01.708764 (+ 14372us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:01.708777 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:01.708827 (+ 50us) server_negotiation.cc:407] Connection header received
0928 14:57:01.708909 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:01.708919 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:01.709260 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:01.713627 (+ 4367us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:01.714270 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:01.716136 (+ 1866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:01.718494 (+ 2358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:01.719593 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:01.720592 (+ 999us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:01.726020 (+ 5428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:01.726051 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:01.726075 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:01.726225 (+ 150us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:01.730467 (+ 4242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:01.731528 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:01.731540 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:01.731544 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:01.731709 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:01.732357 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:01.732361 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:01.732364 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:01.732921 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:01.734149 (+ 1228us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:01.734278 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 14:57:01.734477 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14147,"thread_start_us":15008,"threads_started":1}
I20260928 14:57:01.736407 5839 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:57:01.736536 6379 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 40f51af6fc714868a4c4df423f897b6c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 156238ac7d104bae8996c9dd47abd3cb (127.31.40.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "40f51af6fc714868a4c4df423f897b6c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32909 } } peers { permanent_uuid: "791fc9b073e5437e8af2e1379eb4fe02" member_type: NON_VOTER last_known_addr { host: "127.31.40.61" port: 37667 } attrs { promote: true } } peers { permanent_uuid: "156238ac7d104bae8996c9dd47abd3cb" member_type: NON_VOTER last_known_addr { host: "127.31.40.60" port: 42157 } attrs { promote: true } } }
I20260928 14:57:01.741333 6379 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.31.40.60:42157
I20260928 14:57:01.743448 6266 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [term 1 LEARNER]: Refusing update from remote peer 40f51af6fc714868a4c4df423f897b6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 14:57:01.748230 6219 master.cc:537] Master@127.31.40.60:42157 shutting down...
I20260928 14:57:01.761571 6379 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 40f51af6fc714868a4c4df423f897b6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "791fc9b073e5437e8af2e1379eb4fe02" member_type: NON_VOTER last_known_addr { host: "127.31.40.61" port: 37667 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260928 14:57:01.768168 5641 consensus_peers.cc:487] T 00000000000000000000000000000000 P 40f51af6fc714868a4c4df423f897b6c -> Peer 156238ac7d104bae8996c9dd47abd3cb (127.31.40.60:42157): Couldn't send request to peer 156238ac7d104bae8996c9dd47abd3cb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 14:57:01.772147 6403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40f51af6fc714868a4c4df423f897b6c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "40f51af6fc714868a4c4df423f897b6c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "40f51af6fc714868a4c4df423f897b6c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32909 } } peers { permanent_uuid: "791fc9b073e5437e8af2e1379eb4fe02" member_type: NON_VOTER last_known_addr { host: "127.31.40.61" port: 37667 } attrs { promote: true } } peers { permanent_uuid: "156238ac7d104bae8996c9dd47abd3cb" member_type: NON_VOTER last_known_addr { host: "127.31.40.60" port: 42157 } attrs { promote: true } } }
I20260928 14:57:01.774228 6403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40f51af6fc714868a4c4df423f897b6c [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:01.775252 6266 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 156238ac7d104bae8996c9dd47abd3cb (127.31.40.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "40f51af6fc714868a4c4df423f897b6c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32909 } } peers { permanent_uuid: "791fc9b073e5437e8af2e1379eb4fe02" member_type: NON_VOTER last_known_addr { host: "127.31.40.61" port: 37667 } attrs { promote: true } } peers { permanent_uuid: "156238ac7d104bae8996c9dd47abd3cb" member_type: NON_VOTER last_known_addr { host: "127.31.40.60" port: 42157 } attrs { promote: true } } }
I20260928 14:57:01.781347 6294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "40f51af6fc714868a4c4df423f897b6c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "40f51af6fc714868a4c4df423f897b6c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32909 } } peers { permanent_uuid: "791fc9b073e5437e8af2e1379eb4fe02" member_type: NON_VOTER last_known_addr { host: "127.31.40.61" port: 37667 } attrs { promote: true } } peers { permanent_uuid: "156238ac7d104bae8996c9dd47abd3cb" member_type: NON_VOTER last_known_addr { host: "127.31.40.60" port: 42157 } attrs { promote: true } } }
I20260928 14:57:01.781658 6294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [sys.catalog]: This master's current role is: LEARNER
I20260928 14:57:01.783427 6379 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 40f51af6fc714868a4c4df423f897b6c: attempting to promote NON_VOTER 791fc9b073e5437e8af2e1379eb4fe02 to VOTER
I20260928 14:57:01.784406 6379 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40f51af6fc714868a4c4df423f897b6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "40f51af6fc714868a4c4df423f897b6c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32909 } } peers { permanent_uuid: "791fc9b073e5437e8af2e1379eb4fe02" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 37667 } attrs { promote: false } } peers { permanent_uuid: "156238ac7d104bae8996c9dd47abd3cb" member_type: NON_VOTER last_known_addr { host: "127.31.40.60" port: 42157 } attrs { promote: true } }
I20260928 14:57:01.786789 6266 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [term 1 LEARNER]: Refusing update from remote peer 40f51af6fc714868a4c4df423f897b6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 14:57:01.787417 6404 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 40f51af6fc714868a4c4df423f897b6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "791fc9b073e5437e8af2e1379eb4fe02" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 37667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260928 14:57:01.789106 5641 consensus_peers.cc:487] T 00000000000000000000000000000000 P 40f51af6fc714868a4c4df423f897b6c -> Peer 156238ac7d104bae8996c9dd47abd3cb (127.31.40.60:42157): Couldn't send request to peer 156238ac7d104bae8996c9dd47abd3cb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 14:57:01.791291 6379 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 40f51af6fc714868a4c4df423f897b6c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 791fc9b073e5437e8af2e1379eb4fe02 (127.31.40.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "40f51af6fc714868a4c4df423f897b6c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32909 } } peers { permanent_uuid: "791fc9b073e5437e8af2e1379eb4fe02" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 37667 } attrs { promote: false } } peers { permanent_uuid: "156238ac7d104bae8996c9dd47abd3cb" member_type: NON_VOTER last_known_addr { host: "127.31.40.60" port: 42157 } attrs { promote: true } } }
I20260928 14:57:01.792125 6266 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 791fc9b073e5437e8af2e1379eb4fe02 (127.31.40.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "40f51af6fc714868a4c4df423f897b6c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32909 } } peers { permanent_uuid: "791fc9b073e5437e8af2e1379eb4fe02" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 37667 } attrs { promote: false } } peers { permanent_uuid: "156238ac7d104bae8996c9dd47abd3cb" member_type: NON_VOTER last_known_addr { host: "127.31.40.60" port: 42157 } attrs { promote: true } } }
W20260928 14:57:01.794754 6326 tablet.cc:2369] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:57:01.796386 6294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "40f51af6fc714868a4c4df423f897b6c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "40f51af6fc714868a4c4df423f897b6c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32909 } } peers { permanent_uuid: "791fc9b073e5437e8af2e1379eb4fe02" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 37667 } attrs { promote: false } } peers { permanent_uuid: "156238ac7d104bae8996c9dd47abd3cb" member_type: NON_VOTER last_known_addr { host: "127.31.40.60" port: 42157 } attrs { promote: true } } }
I20260928 14:57:01.796679 6294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:57:01.800554 6219 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb [term 0 FOLLOWER]: Raft consensus shutting down.
I20260928 14:57:01.801043 6219 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb [term 0 FOLLOWER]: Raft consensus is shut down!
I20260928 14:57:01.801339 6219 tablet_replica.cc:330] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb: stopping tablet replica
I20260928 14:57:01.809258 6219 master.cc:552] Master@127.31.40.60:42157 shutdown complete.
I20260928 14:57:01.809342 6219 master_runner.cc:305] Clearing existing system tablet
I20260928 14:57:01.809710 6219 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:01.809808 6219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:01.819047 6403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40f51af6fc714868a4c4df423f897b6c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "40f51af6fc714868a4c4df423f897b6c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "40f51af6fc714868a4c4df423f897b6c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32909 } } peers { permanent_uuid: "791fc9b073e5437e8af2e1379eb4fe02" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 37667 } attrs { promote: false } } peers { permanent_uuid: "156238ac7d104bae8996c9dd47abd3cb" member_type: NON_VOTER last_known_addr { host: "127.31.40.60" port: 42157 } attrs { promote: true } } }
I20260928 14:57:01.819341 6403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40f51af6fc714868a4c4df423f897b6c [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:01.818981 6404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40f51af6fc714868a4c4df423f897b6c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "40f51af6fc714868a4c4df423f897b6c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "40f51af6fc714868a4c4df423f897b6c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32909 } } peers { permanent_uuid: "791fc9b073e5437e8af2e1379eb4fe02" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 37667 } attrs { promote: false } } peers { permanent_uuid: "156238ac7d104bae8996c9dd47abd3cb" member_type: NON_VOTER last_known_addr { host: "127.31.40.60" port: 42157 } attrs { promote: true } } }
I20260928 14:57:01.819525 6404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40f51af6fc714868a4c4df423f897b6c [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:01.819723 6219 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.008s sys 0.002s
I20260928 14:57:01.821524 6411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:01.822013 6219 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:01.822490 6219 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 14:57:01.822643 6219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-2/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-2/wal
uuid: "156238ac7d104bae8996c9dd47abd3cb"
format_stamp: "Formatted at 2026-09-28 14:57:01 on dist-test-slave-j968"
server_key: "f26f1d5ae3f5abc4264aba9561b977e4"
server_key_iv: "6b319ffd65979c5f6f9a2775e6988c01"
server_key_version: "encryptionkey@0"
I20260928 14:57:01.828297 6219 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:57:01.836480 6219 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 14:57:01.836782 6219 log.cc:1196] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 14:57:01.837363 6219 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb: Deleting consensus metadata
I20260928 14:57:01.837679 6219 master_runner.cc:315] Copying system tablet from 127.31.40.62:32909
I20260928 14:57:01.841074 6219 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb: tablet copy: Beginning tablet copy session from remote peer at address 127.31.40.62:32909
I20260928 14:57:01.860524 6188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:01.842070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59396 (local address 127.31.40.62:32909)
0928 14:57:01.842295 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:01.842302 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:01.844309 (+ 2007us) server_negotiation.cc:407] Connection header received
0928 14:57:01.844574 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:01.844580 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:01.844727 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:01.844797 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:01.846097 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:01.847338 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:01.849464 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:01.850140 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:01.850237 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:01.853302 (+ 3065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:01.853309 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:01.853314 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:01.853353 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:01.857248 (+ 3895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:01.858271 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:01.858278 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:01.858282 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:01.858482 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:01.858924 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:01.858929 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:01.858933 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:01.859297 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:01.859438 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:01.859928 (+ 490us) server_negotiation.cc:299] Negotiation successful
0928 14:57:01.860179 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260928 14:57:01.860679 6417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:01.842136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:32909 (local address 127.0.0.1:59396)
0928 14:57:01.843058 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:01.843106 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:01.844431 (+ 1325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:01.845007 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:01.845012 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:01.845649 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:01.845910 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:01.845922 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:01.847537 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:01.847542 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:01.849292 (+ 1750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:01.849307 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:01.850661 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:01.850667 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:01.850856 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:01.851650 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:01.851673 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:01.853061 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:01.857452 (+ 4391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:01.857459 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:01.857463 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:01.858090 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:01.858611 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:01.858617 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:01.858621 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:01.858796 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:01.859418 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:01.859423 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:01.859624 (+ 201us) client_negotiation.cc:769] Sending connection context
0928 14:57:01.859771 (+ 147us) client_negotiation.cc:240] Negotiation successful
0928 14:57:01.860154 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":496,"threads_started":1}
I20260928 14:57:01.861406 5675 tablet_copy_service.cc:140] P 40f51af6fc714868a4c4df423f897b6c: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 156238ac7d104bae8996c9dd47abd3cb ({username='oryx', principal='oryx/127.31.40.60@KRBTEST.COM'} at 127.0.0.1:59396)
I20260928 14:57:01.861593 5675 tablet_copy_service.cc:161] P 40f51af6fc714868a4c4df423f897b6c: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 156238ac7d104bae8996c9dd47abd3cb at {username='oryx', principal='oryx/127.31.40.60@KRBTEST.COM'} at 127.0.0.1:59396: session id = 156238ac7d104bae8996c9dd47abd3cb-00000000000000000000000000000000
I20260928 14:57:01.865773 5675 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 40f51af6fc714868a4c4df423f897b6c: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 14:57:01.867990 6219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 14:57:01.876262 6219 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb: tablet copy: Starting download of 0 data blocks...
I20260928 14:57:01.876556 6219 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb: tablet copy: Starting download of 1 WAL segments...
I20260928 14:57:01.879259 6219 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 14:57:01.884303 5675 tablet_copy_service.cc:342] P 40f51af6fc714868a4c4df423f897b6c: Request end of tablet copy session 156238ac7d104bae8996c9dd47abd3cb-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.31.40.60@KRBTEST.COM'} at 127.0.0.1:59396
I20260928 14:57:01.884447 5675 tablet_copy_service.cc:433] P 40f51af6fc714868a4c4df423f897b6c: ending tablet copy session 156238ac7d104bae8996c9dd47abd3cb-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 156238ac7d104bae8996c9dd47abd3cb
I20260928 14:57:01.896430 6219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:01.899224 6418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:01.905653 6419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:01.910360 6421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:01.912662 6219 server_base.cc:1034] running on GCE node
I20260928 14:57:01.913064 6219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:01.913791 6219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:01.915011 6219 hybrid_clock.cc:648] HybridClock initialized: now 1790607421914968 us; error 35 us; skew 500 ppm
I20260928 14:57:01.915268 6219 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:57:01 dist-test-slave-j968 krb5kdc[5610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607421, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.40.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:01.917478 6219 init.cc:377] Logged in from keytab as oryx/127.31.40.60@KRBTEST.COM (short username oryx)
I20260928 14:57:01.918828 6219 webserver.cc:460] Webserver started at http://127.31.40.60:34559/ using document root <none> and password file <none>
I20260928 14:57:01.919139 6219 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:01.919317 6219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:01.928578 6219 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.000s sys 0.008s
I20260928 14:57:01.929790 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 5716
I20260928 14:57:01.931547 6428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:01.932668 6219 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260928 14:57:01.932804 6219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-2/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-2/wal
uuid: "156238ac7d104bae8996c9dd47abd3cb"
format_stamp: "Formatted at 2026-09-28 14:57:01 on dist-test-slave-j968"
server_key: "f26f1d5ae3f5abc4264aba9561b977e4"
server_key_iv: "6b319ffd65979c5f6f9a2775e6988c01"
server_key_version: "encryptionkey@0"
I20260928 14:57:01.933022 6219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:01.947053 6219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:01.951689 6219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:01.955289 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.1:35061
--local_ip_for_outbound_sockets=127.31.40.1
--tserver_master_addrs=127.31.40.62:32909
--webserver_port=40027
--webserver_interface=127.31.40.1
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/krb5kdc/oryx_127.31.40.1.keytab
--principal=oryx/127.31.40.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:32961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.31.40.62:32909,127.31.40.61:37667
--tserver_master_addrs=127.31.40.62:32909,127.31.40.61:37667,127.31.40.60:42157 with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:57:01.978636 6219 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.60:42157
I20260928 14:57:01.978678 6480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.60:42157 every 8 connection(s)
I20260928 14:57:01.979890 6219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/master-2/data/info.pb
I20260928 14:57:01.995445 6481 sys_catalog.cc:263] Verifying existing consensus state
I20260928 14:57:01.999310 6481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb: Bootstrap starting.
I20260928 14:57:02.018466 6481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:57:02.019001 6481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb: Bootstrap complete.
I20260928 14:57:02.020162 6481 raft_consensus.cc:348] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "40f51af6fc714868a4c4df423f897b6c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32909 } } peers { permanent_uuid: "791fc9b073e5437e8af2e1379eb4fe02" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 37667 } attrs { promote: false } } peers { permanent_uuid: "156238ac7d104bae8996c9dd47abd3cb" member_type: NON_VOTER last_known_addr { host: "127.31.40.60" port: 42157 } attrs { promote: true } }
I20260928 14:57:02.020483 6481 raft_consensus.cc:729] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 156238ac7d104bae8996c9dd47abd3cb, State: Initialized, Role: LEARNER
I20260928 14:57:02.020766 6481 consensus_queue.cc:260] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "40f51af6fc714868a4c4df423f897b6c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32909 } } peers { permanent_uuid: "791fc9b073e5437e8af2e1379eb4fe02" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 37667 } attrs { promote: false } } peers { permanent_uuid: "156238ac7d104bae8996c9dd47abd3cb" member_type: NON_VOTER last_known_addr { host: "127.31.40.60" port: 42157 } attrs { promote: true } }
I20260928 14:57:02.021919 6481 sys_catalog.cc:564] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:57:02.021639 6484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "40f51af6fc714868a4c4df423f897b6c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32909 } } peers { permanent_uuid: "791fc9b073e5437e8af2e1379eb4fe02" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 37667 } attrs { promote: false } } peers { permanent_uuid: "156238ac7d104bae8996c9dd47abd3cb" member_type: NON_VOTER last_known_addr { host: "127.31.40.60" port: 42157 } attrs { promote: true } } }
I20260928 14:57:02.022262 6484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb [sys.catalog]: This master's current role is: LEARNER
I20260928 14:57:02.030639 6495 catalog_manager.cc:1259] Loaded cluster ID: 913af42657514f2aa51a685dba5d7ba1
I20260928 14:57:02.030822 6495 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb: loading cluster ID for follower catalog manager: success
I20260928 14:57:02.034919 6495 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb: acquiring CA information for follower catalog manager: success
I20260928 14:57:02.037591 6495 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 14:57:02.238909 6496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:02.232155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45172 (local address 127.31.40.60:42157)
0928 14:57:02.233104 (+ 949us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:02.233109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:02.233130 (+ 21us) server_negotiation.cc:407] Connection header received
0928 14:57:02.233194 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:02.233198 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:02.233267 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:02.233400 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:02.233916 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:02.235306 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:02.236979 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:02.237818 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:02.237982 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:02.238235 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:02.238428 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 14:57:02.238573 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":756,"thread_start_us":456,"threads_started":1}
I20260928 14:57:02.238914 6380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:02.231922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:42157 (local address 127.0.0.1:45172)
0928 14:57:02.232166 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:02.232187 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:02.232341 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:02.233437 (+ 1096us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:02.233442 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:02.233486 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:02.233716 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:02.233728 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:02.235521 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:02.235528 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:02.236795 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:02.236811 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:02.238002 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:02.238009 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:02.238226 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:02.238270 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 14:57:02.238394 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 14:57:02.238565 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260928 14:57:02.240172 6456 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260928 14:57:02.240934 6484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40f51af6fc714868a4c4df423f897b6c. Latest consensus state: current_term: 1 leader_uuid: "40f51af6fc714868a4c4df423f897b6c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "40f51af6fc714868a4c4df423f897b6c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32909 } } peers { permanent_uuid: "791fc9b073e5437e8af2e1379eb4fe02" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 37667 } attrs { promote: false } } peers { permanent_uuid: "156238ac7d104bae8996c9dd47abd3cb" member_type: NON_VOTER last_known_addr { host: "127.31.40.60" port: 42157 } attrs { promote: true } } }
I20260928 14:57:02.241245 6484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb [sys.catalog]: This master's current role is: LEARNER
W20260928 14:57:02.356762 6446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:02.357151 6446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:02.357352 6446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:02.367856 6446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:02.368055 6446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:02.368186 6446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.1
I20260928 14:57:02.383610 6446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:32961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/krb5kdc/oryx_127.31.40.1.keytab
--principal=oryx/127.31.40.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.1:35061
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.40.1
--webserver_port=40027
--tserver_master_addrs=127.31.40.62:32909,127.31.40.61:37667,127.31.40.60:42157
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.1
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:57:02.385221 6446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:02.386955 6446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:02.403000 6501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:02.402889 6504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:02.402858 6502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:02.403815 6446 server_base.cc:1034] running on GCE node
I20260928 14:57:02.404670 6446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:02.405921 6446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:02.407209 6446 hybrid_clock.cc:648] HybridClock initialized: now 1790607422407081 us; error 38 us; skew 500 ppm
I20260928 14:57:02.407667 6446 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:57:02 dist-test-slave-j968 krb5kdc[5610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607422, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.40.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:02.411206 6446 init.cc:377] Logged in from keytab as oryx/127.31.40.1@KRBTEST.COM (short username oryx)
I20260928 14:57:02.412904 6446 webserver.cc:460] Webserver started at http://127.31.40.1:40027/ using document root <none> and password file <none>
I20260928 14:57:02.413640 6446 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:02.413765 6446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:02.419802 6446 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 14:57:02.423058 6511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:02.425057 6446 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 14:57:02.425279 6446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-0/wal
uuid: "50bbf97992ac4d92a0ce12b7b1be3bbe"
format_stamp: "Formatted at 2026-09-28 14:56:59 on dist-test-slave-j968"
server_key: "26442481750400b0e779d3b7da95122f"
server_key_iv: "896fbc2040a586b1d715c9dcfb3287ad"
server_key_version: "encryptionkey@0"
I20260928 14:57:02.426612 6446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:02.444592 6446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:02.450696 6446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:02.451210 6446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:02.452420 6446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:57:02.454157 6446 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:57:02.454250 6446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:02.454399 6446 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:57:02.454506 6446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:02.500252 6446 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.1:35061
I20260928 14:57:02.500361 6623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.1:35061 every 8 connection(s)
I20260928 14:57:02.501916 6446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
I20260928 14:57:02.509155 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 6446
I20260928 14:57:02.509653 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 5850
Sep 28 14:57:02 dist-test-slave-j968 krb5kdc[5610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607422, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.40.1@KRBTEST.COM for oryx/127.31.40.61@KRBTEST.COM
I20260928 14:57:02.545150 6628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:02.505203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:45029 (local address 127.31.40.61:37667)
0928 14:57:02.520406 (+ 15203us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:02.520415 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:02.520831 (+ 416us) server_negotiation.cc:407] Connection header received
0928 14:57:02.523478 (+ 2647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:02.523483 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:02.523598 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:02.523716 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:02.526403 (+ 2687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:02.527265 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:02.529975 (+ 2710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:02.530547 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:02.530703 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:02.536457 (+ 5754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:02.536465 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:02.536470 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:02.536517 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:02.540690 (+ 4173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:02.541598 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:02.541605 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:02.541610 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:02.541774 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:02.542506 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:02.542518 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:02.542520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:02.542933 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:02.543090 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:02.544443 (+ 1353us) server_negotiation.cc:299] Negotiation successful
0928 14:57:02.544803 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15036,"thread_start_us":15220,"threads_started":1}
I20260928 14:57:02.547360 6631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:02.506025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.61:37667 (local address 127.31.40.1:45029)
0928 14:57:02.520629 (+ 14604us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:02.520682 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:02.523089 (+ 2407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:02.524268 (+ 1179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:02.524280 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:02.525220 (+ 940us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:02.526073 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:02.526107 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:02.527468 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:02.527474 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:02.529786 (+ 2312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:02.529800 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:02.530819 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:02.530829 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:02.531175 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:02.532096 (+ 921us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:02.532121 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:02.536110 (+ 3989us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:02.540891 (+ 4781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:02.540902 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:02.540930 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:02.541439 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:02.541980 (+ 541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:02.541985 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:02.541989 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:02.542321 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:02.543210 (+ 889us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:02.543225 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:02.543848 (+ 623us) client_negotiation.cc:769] Sending connection context
0928 14:57:02.545519 (+ 1671us) client_negotiation.cc:240] Negotiation successful
0928 14:57:02.546015 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14206,"spinlock_wait_cycles":6528,"thread_start_us":26619,"threads_started":1}
Sep 28 14:57:02 dist-test-slave-j968 krb5kdc[5610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607422, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.40.1@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM
Sep 28 14:57:02 dist-test-slave-j968 krb5kdc[5610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.40.2@KRBTEST.COM
I20260928 14:57:02.551337 6626 heartbeater.cc:347] Connected to a master server at 127.31.40.61:37667
I20260928 14:57:02.551896 6626 heartbeater.cc:464] Registering TS with master...
Sep 28 14:57:02 dist-test-slave-j968 krb5kdc[5610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607422, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.40.1@KRBTEST.COM for oryx/127.31.40.60@KRBTEST.COM
I20260928 14:57:02.553190 6626 heartbeater.cc:511] Master 127.31.40.61:37667 requested a full tablet report, sending...
I20260928 14:57:02.556229 6256 ts_manager.cc:194] Registered new tserver with Master: 50bbf97992ac4d92a0ce12b7b1be3bbe (127.31.40.1:35061)
I20260928 14:57:02.557258 6630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:02.506019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:32909 (local address 127.31.40.1:52803)
0928 14:57:02.532671 (+ 26652us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:02.532708 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:02.532849 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:02.533940 (+ 1091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:02.533946 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:02.534612 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:02.535383 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:02.535405 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:02.537712 (+ 2307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:02.537719 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:02.540694 (+ 2975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:02.540713 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:02.541979 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:02.541986 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:02.542173 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:02.542946 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:02.542973 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:02.548664 (+ 5691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:02.553803 (+ 5139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:02.553809 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:02.553813 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:02.554480 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:02.555099 (+ 619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:02.555104 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:02.555107 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:02.555266 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:02.556012 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:02.556025 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:02.556203 (+ 178us) client_negotiation.cc:769] Sending connection context
0928 14:57:02.556367 (+ 164us) client_negotiation.cc:240] Negotiation successful
0928 14:57:02.556708 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26461,"mutex_wait_us":42,"spinlock_wait_cycles":93952,"thread_start_us":17282,"threads_started":1}
I20260928 14:57:02.558364 6629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:02.505263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:52803 (local address 127.31.40.62:32909)
0928 14:57:02.529092 (+ 23829us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:02.529100 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:02.533046 (+ 3946us) server_negotiation.cc:407] Connection header received
0928 14:57:02.533129 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:02.533137 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:02.533278 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:02.533503 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:02.535644 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:02.537538 (+ 1894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:02.540854 (+ 3316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:02.541711 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:02.541905 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:02.549117 (+ 7212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:02.549128 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:02.549136 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:02.549205 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:02.553450 (+ 4245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:02.554658 (+ 1208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:02.554667 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:02.554672 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:02.554870 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:02.555412 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:02.555418 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:02.555422 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:02.555820 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:02.556027 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:02.557817 (+ 1790us) server_negotiation.cc:299] Negotiation successful
0928 14:57:02.558023 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23688,"thread_start_us":23560,"threads_started":1}
I20260928 14:57:02.559441 6625 heartbeater.cc:347] Connected to a master server at 127.31.40.62:32909
I20260928 14:57:02.559617 6625 heartbeater.cc:464] Registering TS with master...
I20260928 14:57:02.560299 6625 heartbeater.cc:511] Master 127.31.40.62:32909 requested a full tablet report, sending...
I20260928 14:57:02.561743 5655 ts_manager.cc:194] Re-registered known tserver with Master: 50bbf97992ac4d92a0ce12b7b1be3bbe (127.31.40.1:35061)
I20260928 14:57:02.562218 6496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:02.533743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:49121 (local address 127.31.40.60:42157)
0928 14:57:02.533942 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:02.533949 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:02.539492 (+ 5543us) server_negotiation.cc:407] Connection header received
0928 14:57:02.539798 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:02.539807 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:02.539971 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:02.540068 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:02.543125 (+ 3057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:02.544448 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:02.546960 (+ 2512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:02.547740 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:02.547917 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:02.553874 (+ 5957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:02.553885 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:02.553892 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:02.553940 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:02.558176 (+ 4236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:02.559526 (+ 1350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:02.559533 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:02.559536 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:02.559677 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:02.560219 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:02.560225 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:02.560229 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:02.560584 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:02.560737 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:02.561429 (+ 692us) server_negotiation.cc:299] Negotiation successful
0928 14:57:02.561751 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 14:57:02.562444 6633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:02.510888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42932 (local address 127.31.40.2:33025)
0928 14:57:02.529527 (+ 18639us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:02.529536 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:02.529566 (+ 30us) server_negotiation.cc:407] Connection header received
0928 14:57:02.529627 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:02.529633 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:02.529885 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:02.530111 (+ 226us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:02.532893 (+ 2782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:02.534449 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:02.536704 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:02.538596 (+ 1892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:02.538772 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:02.551779 (+ 13007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:02.551806 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:02.551825 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:02.551904 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:02.558343 (+ 6439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:02.559490 (+ 1147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:02.559500 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:02.559503 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:02.559673 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:02.560245 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:02.560252 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:02.560255 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:02.560759 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:02.561103 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:02.561658 (+ 555us) server_negotiation.cc:299] Negotiation successful
0928 14:57:02.561978 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18473,"thread_start_us":19442,"threads_started":1}
I20260928 14:57:02.562947 5655 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.40.1@KRBTEST.COM'} at 127.31.40.1:52803
I20260928 14:57:02.563097 6634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:02.534668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:42157 (local address 127.31.40.1:49121)
0928 14:57:02.539371 (+ 4703us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:02.539396 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:02.539618 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:02.540314 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:02.540320 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:02.542433 (+ 2113us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:02.542878 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:02.542895 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:02.544654 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:02.544663 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:02.546768 (+ 2105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:02.546788 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:02.548712 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:02.548721 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:02.549003 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:02.550061 (+ 1058us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:02.550085 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:02.553519 (+ 3434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:02.558608 (+ 5089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:02.558613 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:02.558616 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:02.559361 (+ 745us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:02.559833 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:02.559841 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:02.559845 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:02.560079 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:02.560874 (+ 795us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:02.560877 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:02.561047 (+ 170us) client_negotiation.cc:769] Sending connection context
0928 14:57:02.562315 (+ 1268us) client_negotiation.cc:240] Negotiation successful
0928 14:57:02.562685 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4541,"thread_start_us":3438,"threads_started":1}
I20260928 14:57:02.564225 5973 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:57:02.564644 6624 heartbeater.cc:347] Connected to a master server at 127.31.40.60:42157
I20260928 14:57:02.564797 6624 heartbeater.cc:464] Registering TS with master...
I20260928 14:57:02.565601 6624 heartbeater.cc:511] Master 127.31.40.60:42157 requested a full tablet report, sending...
I20260928 14:57:02.568864 6445 ts_manager.cc:194] Registered new tserver with Master: 50bbf97992ac4d92a0ce12b7b1be3bbe (127.31.40.1:35061)
I20260928 14:57:02.656412 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 5850
I20260928 14:57:02.674046 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.2:33025
--local_ip_for_outbound_sockets=127.31.40.2
--tserver_master_addrs=127.31.40.62:32909
--webserver_port=41141
--webserver_interface=127.31.40.2
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/krb5kdc/oryx_127.31.40.2.keytab
--principal=oryx/127.31.40.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:32961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.31.40.62:32909,127.31.40.61:37667
--tserver_master_addrs=127.31.40.62:32909,127.31.40.61:37667,127.31.40.60:42157 with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:57:02.801281 6404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40f51af6fc714868a4c4df423f897b6c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "40f51af6fc714868a4c4df423f897b6c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "40f51af6fc714868a4c4df423f897b6c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32909 } } peers { permanent_uuid: "791fc9b073e5437e8af2e1379eb4fe02" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 37667 } attrs { promote: false } } peers { permanent_uuid: "156238ac7d104bae8996c9dd47abd3cb" member_type: NON_VOTER last_known_addr { host: "127.31.40.60" port: 42157 } attrs { promote: true } } }
I20260928 14:57:02.801674 6404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40f51af6fc714868a4c4df423f897b6c [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:02.803203 6404 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 40f51af6fc714868a4c4df423f897b6c: attempting to promote NON_VOTER 156238ac7d104bae8996c9dd47abd3cb to VOTER
I20260928 14:57:02.803802 6404 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40f51af6fc714868a4c4df423f897b6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "40f51af6fc714868a4c4df423f897b6c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32909 } } peers { permanent_uuid: "791fc9b073e5437e8af2e1379eb4fe02" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 37667 } attrs { promote: false } } peers { permanent_uuid: "156238ac7d104bae8996c9dd47abd3cb" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 42157 } attrs { promote: false } }
I20260928 14:57:02.806242 6266 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [term 1 FOLLOWER]: Refusing update from remote peer 40f51af6fc714868a4c4df423f897b6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260928 14:57:02.806486 6456 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb [term 1 LEARNER]: Refusing update from remote peer 40f51af6fc714868a4c4df423f897b6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260928 14:57:02.807009 6404 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 40f51af6fc714868a4c4df423f897b6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "791fc9b073e5437e8af2e1379eb4fe02" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 37667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260928 14:57:02.807415 6638 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 40f51af6fc714868a4c4df423f897b6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "156238ac7d104bae8996c9dd47abd3cb" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 42157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260928 14:57:02.810499 6404 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 40f51af6fc714868a4c4df423f897b6c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 156238ac7d104bae8996c9dd47abd3cb (127.31.40.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "40f51af6fc714868a4c4df423f897b6c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32909 } } peers { permanent_uuid: "791fc9b073e5437e8af2e1379eb4fe02" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 37667 } attrs { promote: false } } peers { permanent_uuid: "156238ac7d104bae8996c9dd47abd3cb" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 42157 } attrs { promote: false } } }
I20260928 14:57:02.811245 6266 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 156238ac7d104bae8996c9dd47abd3cb (127.31.40.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "40f51af6fc714868a4c4df423f897b6c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32909 } } peers { permanent_uuid: "791fc9b073e5437e8af2e1379eb4fe02" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 37667 } attrs { promote: false } } peers { permanent_uuid: "156238ac7d104bae8996c9dd47abd3cb" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 42157 } attrs { promote: false } } }
I20260928 14:57:02.815907 6638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40f51af6fc714868a4c4df423f897b6c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "40f51af6fc714868a4c4df423f897b6c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "40f51af6fc714868a4c4df423f897b6c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32909 } } peers { permanent_uuid: "791fc9b073e5437e8af2e1379eb4fe02" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 37667 } attrs { promote: false } } peers { permanent_uuid: "156238ac7d104bae8996c9dd47abd3cb" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 42157 } attrs { promote: false } } }
I20260928 14:57:02.815933 6404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40f51af6fc714868a4c4df423f897b6c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "40f51af6fc714868a4c4df423f897b6c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "40f51af6fc714868a4c4df423f897b6c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32909 } } peers { permanent_uuid: "791fc9b073e5437e8af2e1379eb4fe02" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 37667 } attrs { promote: false } } peers { permanent_uuid: "156238ac7d104bae8996c9dd47abd3cb" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 42157 } attrs { promote: false } } }
I20260928 14:57:02.816277 6638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40f51af6fc714868a4c4df423f897b6c [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:02.816303 6404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40f51af6fc714868a4c4df423f897b6c [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:02.816497 6404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40f51af6fc714868a4c4df423f897b6c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "40f51af6fc714868a4c4df423f897b6c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "40f51af6fc714868a4c4df423f897b6c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32909 } } peers { permanent_uuid: "791fc9b073e5437e8af2e1379eb4fe02" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 37667 } attrs { promote: false } } peers { permanent_uuid: "156238ac7d104bae8996c9dd47abd3cb" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 42157 } attrs { promote: false } } }
I20260928 14:57:02.816888 6404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40f51af6fc714868a4c4df423f897b6c [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:02.816039 6456 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 156238ac7d104bae8996c9dd47abd3cb (127.31.40.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "40f51af6fc714868a4c4df423f897b6c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32909 } } peers { permanent_uuid: "791fc9b073e5437e8af2e1379eb4fe02" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 37667 } attrs { promote: false } } peers { permanent_uuid: "156238ac7d104bae8996c9dd47abd3cb" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 42157 } attrs { promote: false } } }
I20260928 14:57:02.820187 6645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "40f51af6fc714868a4c4df423f897b6c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "40f51af6fc714868a4c4df423f897b6c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32909 } } peers { permanent_uuid: "791fc9b073e5437e8af2e1379eb4fe02" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 37667 } attrs { promote: false } } peers { permanent_uuid: "156238ac7d104bae8996c9dd47abd3cb" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 42157 } attrs { promote: false } } }
I20260928 14:57:02.820494 6645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:57:02.821094 6643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "40f51af6fc714868a4c4df423f897b6c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "40f51af6fc714868a4c4df423f897b6c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32909 } } peers { permanent_uuid: "791fc9b073e5437e8af2e1379eb4fe02" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 37667 } attrs { promote: false } } peers { permanent_uuid: "156238ac7d104bae8996c9dd47abd3cb" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 42157 } attrs { promote: false } } }
I20260928 14:57:02.821532 6643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 14:57:03.051613 6636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:03.052011 6636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:03.052174 6636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:03.060676 6636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:03.060797 6636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:03.060978 6636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.2
I20260928 14:57:03.071919 6636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:32961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/krb5kdc/oryx_127.31.40.2.keytab
--principal=oryx/127.31.40.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.2:33025
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.40.2
--webserver_port=41141
--tserver_master_addrs=127.31.40.62:32909,127.31.40.61:37667,127.31.40.60:42157
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.2
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:57:03.073400 6636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:03.074859 6636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:03.089404 6651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:03.089468 6650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:03.089493 6653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:03.090310 6636 server_base.cc:1034] running on GCE node
I20260928 14:57:03.091071 6636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:03.092290 6636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:03.093595 6636 hybrid_clock.cc:648] HybridClock initialized: now 1790607423093547 us; error 33 us; skew 500 ppm
I20260928 14:57:03.094017 6636 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:57:03 dist-test-slave-j968 krb5kdc[5610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607423, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.40.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:03.097802 6636 init.cc:377] Logged in from keytab as oryx/127.31.40.2@KRBTEST.COM (short username oryx)
I20260928 14:57:03.099597 6636 webserver.cc:460] Webserver started at http://127.31.40.2:41141/ using document root <none> and password file <none>
I20260928 14:57:03.100432 6636 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:03.100585 6636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:03.106557 6636 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 14:57:03.109835 6660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:03.111605 6636 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 14:57:03.111842 6636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-1/wal
uuid: "1484aa0f119348a1af94cea1a988d0f2"
format_stamp: "Formatted at 2026-09-28 14:56:59 on dist-test-slave-j968"
server_key: "dfa0b50c70221f6722dddc8f540b7a74"
server_key_iv: "e46fa74b10ab334af7fb216ef7a49bf2"
server_key_version: "encryptionkey@0"
I20260928 14:57:03.112957 6636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:03.135198 6636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:03.140228 6636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:03.140650 6636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:03.141806 6636 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:57:03.143709 6636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:57:03.143807 6636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:03.144023 6636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:57:03.144130 6636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:03.178864 6636 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.2:33025
I20260928 14:57:03.179066 6772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.2:33025 every 8 connection(s)
I20260928 14:57:03.180619 6636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
I20260928 14:57:03.181752 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 6636
I20260928 14:57:03.182138 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 5984
Sep 28 14:57:03 dist-test-slave-j968 krb5kdc[5610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.40.3@KRBTEST.COM
Sep 28 14:57:03 dist-test-slave-j968 krb5kdc[5610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607423, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.40.2@KRBTEST.COM for oryx/127.31.40.61@KRBTEST.COM
Sep 28 14:57:03 dist-test-slave-j968 krb5kdc[5610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607423, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.40.2@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM
Sep 28 14:57:03 dist-test-slave-j968 krb5kdc[5610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607423, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.40.2@KRBTEST.COM for oryx/127.31.40.60@KRBTEST.COM
I20260928 14:57:03.236675 6778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:03.183240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32798 (local address 127.31.40.3:34255)
0928 14:57:03.210953 (+ 27713us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:03.210967 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:03.211706 (+ 739us) server_negotiation.cc:407] Connection header received
0928 14:57:03.211939 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:03.211964 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:03.212356 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:03.212598 (+ 242us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:03.215835 (+ 3237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:03.217524 (+ 1689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:03.218883 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:03.219702 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:03.219856 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:03.227065 (+ 7209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:03.227081 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:03.227098 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:03.227139 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:03.231087 (+ 3948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:03.232725 (+ 1638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:03.232739 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:03.232746 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:03.232925 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:03.233457 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:03.233462 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:03.233465 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:03.234107 (+ 642us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:03.234422 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:03.235908 (+ 1486us) server_negotiation.cc:299] Negotiation successful
0928 14:57:03.236218 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27505,"thread_start_us":27345,"threads_started":1}
I20260928 14:57:03.238235 6107 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:57:03.240782 6779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:03.183821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:48465 (local address 127.31.40.62:32909)
0928 14:57:03.215821 (+ 32000us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:03.215831 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:03.215853 (+ 22us) server_negotiation.cc:407] Connection header received
0928 14:57:03.215945 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:03.215954 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:03.216066 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:03.216218 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:03.219083 (+ 2865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:03.220512 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:03.222819 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:03.223569 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:03.223716 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:03.227719 (+ 4003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:03.227728 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:03.227734 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:03.227792 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:03.232936 (+ 5144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:03.234272 (+ 1336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:03.234277 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:03.234280 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:03.234444 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:03.236020 (+ 1576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:03.236027 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:03.236033 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:03.236362 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:03.236485 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:03.240079 (+ 3594us) server_negotiation.cc:299] Negotiation successful
0928 14:57:03.240408 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31827,"thread_start_us":31673,"threads_started":1}
I20260928 14:57:03.241120 6784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:03.209850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:42157 (local address 127.31.40.2:47549)
0928 14:57:03.221721 (+ 11871us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:03.221744 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:03.221919 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:03.222491 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:03.222496 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:03.223079 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:03.223398 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:03.223409 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:03.225076 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:03.225083 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:03.228018 (+ 2935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:03.228031 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:03.229126 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:03.229133 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:03.229335 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:03.229929 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:03.229952 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:03.232662 (+ 2710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:03.237239 (+ 4577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:03.237245 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:03.237249 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:03.237616 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:03.238055 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:03.238061 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:03.238065 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:03.238217 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:03.238899 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:03.238906 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:03.239089 (+ 183us) client_negotiation.cc:769] Sending connection context
0928 14:57:03.239247 (+ 158us) client_negotiation.cc:240] Negotiation successful
0928 14:57:03.239590 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11663,"spinlock_wait_cycles":23936,"thread_start_us":11636,"threads_started":1}
I20260928 14:57:03.241727 6780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:03.184741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:32909 (local address 127.31.40.2:48465)
0928 14:57:03.197957 (+ 13216us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:03.198025 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:03.201301 (+ 3276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:03.216276 (+ 14975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:03.216282 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:03.216812 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:03.217108 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:03.217120 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:03.220661 (+ 3541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:03.220670 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:03.222574 (+ 1904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:03.222594 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:03.223779 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:03.223785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:03.224023 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:03.224719 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:03.224747 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:03.227447 (+ 2700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:03.233110 (+ 5663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:03.233120 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:03.233139 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:03.234098 (+ 959us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:03.234593 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:03.234648 (+ 55us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:03.234652 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:03.234898 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:03.236574 (+ 1676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:03.236583 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:03.238165 (+ 1582us) client_negotiation.cc:769] Sending connection context
0928 14:57:03.238500 (+ 335us) client_negotiation.cc:240] Negotiation successful
0928 14:57:03.239432 (+ 932us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12808,"thread_start_us":11006,"threads_started":1}
I20260928 14:57:03.242297 6783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:03.204894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:47549 (local address 127.31.40.60:42157)
0928 14:57:03.221119 (+ 16225us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:03.221126 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:03.222049 (+ 923us) server_negotiation.cc:407] Connection header received
0928 14:57:03.222119 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:03.222125 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:03.222202 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:03.222300 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:03.223582 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:03.224911 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:03.228189 (+ 3278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:03.228893 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:03.229039 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:03.232897 (+ 3858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:03.232904 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:03.232908 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:03.232949 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:03.237098 (+ 4149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:03.237798 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:03.237807 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:03.237811 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:03.237956 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:03.238342 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:03.238348 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:03.238351 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:03.238752 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:03.241563 (+ 2811us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:03.241685 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 14:57:03.241959 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16033,"thread_start_us":16486,"threads_started":1}
I20260928 14:57:03.243774 6773 heartbeater.cc:347] Connected to a master server at 127.31.40.60:42157
I20260928 14:57:03.244115 6773 heartbeater.cc:464] Registering TS with master...
I20260928 14:57:03.244407 6782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:03.190278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:53247 (local address 127.31.40.61:37667)
0928 14:57:03.205075 (+ 14797us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:03.205082 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:03.205110 (+ 28us) server_negotiation.cc:407] Connection header received
0928 14:57:03.205220 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:03.205228 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:03.205342 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:03.205477 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:03.208950 (+ 3473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:03.210233 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:03.216469 (+ 6236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:03.217413 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:03.217560 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:03.228032 (+ 10472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:03.228039 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:03.228043 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:03.228105 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:03.231941 (+ 3836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:03.241231 (+ 9290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:03.241238 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:03.241242 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:03.241399 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:03.242751 (+ 1352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:03.242757 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:03.242761 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:03.243187 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:03.243353 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:03.243954 (+ 601us) server_negotiation.cc:299] Negotiation successful
0928 14:57:03.244184 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14659,"thread_start_us":18396,"threads_started":1}
I20260928 14:57:03.244925 6773 heartbeater.cc:511] Master 127.31.40.60:42157 requested a full tablet report, sending...
I20260928 14:57:03.245096 6781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:03.189582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.61:37667 (local address 127.31.40.2:53247)
0928 14:57:03.204497 (+ 14915us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:03.204524 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:03.204653 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:03.206158 (+ 1505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:03.206177 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:03.207413 (+ 1236us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:03.208672 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:03.208704 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:03.212318 (+ 3614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:03.212328 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:03.215185 (+ 2857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:03.215212 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:03.218092 (+ 2880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:03.218102 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:03.218529 (+ 427us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:03.220888 (+ 2359us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:03.220921 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:03.227770 (+ 6849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:03.235301 (+ 7531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:03.235306 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:03.235309 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:03.235830 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:03.242427 (+ 6597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:03.242432 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:03.242435 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:03.242592 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:03.243443 (+ 851us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:03.243447 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:03.243607 (+ 160us) client_negotiation.cc:769] Sending connection context
0928 14:57:03.244543 (+ 936us) client_negotiation.cc:240] Negotiation successful
0928 14:57:03.244770 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14751,"thread_start_us":14535,"threads_started":1}
I20260928 14:57:03.245747 6774 heartbeater.cc:347] Connected to a master server at 127.31.40.62:32909
I20260928 14:57:03.245889 6774 heartbeater.cc:464] Registering TS with master...
I20260928 14:57:03.246202 6774 heartbeater.cc:511] Master 127.31.40.62:32909 requested a full tablet report, sending...
I20260928 14:57:03.247391 6775 heartbeater.cc:347] Connected to a master server at 127.31.40.61:37667
I20260928 14:57:03.247529 6775 heartbeater.cc:464] Registering TS with master...
I20260928 14:57:03.247611 6445 ts_manager.cc:194] Registered new tserver with Master: 1484aa0f119348a1af94cea1a988d0f2 (127.31.40.2:33025)
I20260928 14:57:03.247951 6775 heartbeater.cc:511] Master 127.31.40.61:37667 requested a full tablet report, sending...
I20260928 14:57:03.248323 5655 ts_manager.cc:194] Re-registered known tserver with Master: 1484aa0f119348a1af94cea1a988d0f2 (127.31.40.2:33025)
I20260928 14:57:03.249229 6256 ts_manager.cc:194] Registered new tserver with Master: 1484aa0f119348a1af94cea1a988d0f2 (127.31.40.2:33025)
I20260928 14:57:03.249327 5655 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.40.2@KRBTEST.COM'} at 127.31.40.2:48465
I20260928 14:57:03.335655 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 5984
I20260928 14:57:03.353582 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.3:34255
--local_ip_for_outbound_sockets=127.31.40.3
--tserver_master_addrs=127.31.40.62:32909
--webserver_port=38755
--webserver_interface=127.31.40.3
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/krb5kdc/oryx_127.31.40.3.keytab
--principal=oryx/127.31.40.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:32961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.31.40.62:32909,127.31.40.61:37667
--tserver_master_addrs=127.31.40.62:32909,127.31.40.61:37667,127.31.40.60:42157 with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:57:03.566902 6625 heartbeater.cc:503] Master 127.31.40.62:32909 was elected leader, sending a full tablet report...
W20260928 14:57:03.702224 6786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:03.702569 6786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:03.702741 6786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:03.710441 6786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:03.710616 6786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:03.710805 6786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.3
I20260928 14:57:03.721540 6786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:32961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/krb5kdc/oryx_127.31.40.3.keytab
--principal=oryx/127.31.40.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.3:34255
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.40.3
--webserver_port=38755
--tserver_master_addrs=127.31.40.62:32909,127.31.40.61:37667,127.31.40.60:42157
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.3
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:57:03.723133 6786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:03.724687 6786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:03.738962 6794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:03.739029 6792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:03.739001 6791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:03.740222 6786 server_base.cc:1034] running on GCE node
I20260928 14:57:03.740998 6786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:03.742328 6786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:03.743654 6786 hybrid_clock.cc:648] HybridClock initialized: now 1790607423743547 us; error 68 us; skew 500 ppm
I20260928 14:57:03.744328 6786 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:57:03 dist-test-slave-j968 krb5kdc[5610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607423, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.40.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:03.748154 6786 init.cc:377] Logged in from keytab as oryx/127.31.40.3@KRBTEST.COM (short username oryx)
I20260928 14:57:03.749898 6786 webserver.cc:460] Webserver started at http://127.31.40.3:38755/ using document root <none> and password file <none>
I20260928 14:57:03.750675 6786 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:03.750803 6786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:03.757159 6786 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 14:57:03.760393 6801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:03.761986 6786 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 14:57:03.762197 6786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-2/wal
uuid: "f2f0f28f0299472386abafe15704df7b"
format_stamp: "Formatted at 2026-09-28 14:57:00 on dist-test-slave-j968"
server_key: "2feb37b4657d920bbb6dd87a78119f3d"
server_key_iv: "8a57bdf247cd5a31be088d8bf64f6dfc"
server_key_version: "encryptionkey@0"
I20260928 14:57:03.763355 6786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:03.781013 6786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:03.785606 6786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:03.785914 6786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:03.786965 6786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:57:03.788626 6786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:57:03.788748 6786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:03.788884 6786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:57:03.788975 6786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:03.825070 6786 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.3:34255
I20260928 14:57:03.825237 6913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.3:34255 every 8 connection(s)
I20260928 14:57:03.826813 6786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
I20260928 14:57:03.836336 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 6786
Sep 28 14:57:03 dist-test-slave-j968 krb5kdc[5610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607423, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.40.3@KRBTEST.COM for oryx/127.31.40.60@KRBTEST.COM
Sep 28 14:57:03 dist-test-slave-j968 krb5kdc[5610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607423, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.40.3@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM
Sep 28 14:57:03 dist-test-slave-j968 krb5kdc[5610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607423, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.40.3@KRBTEST.COM for oryx/127.31.40.61@KRBTEST.COM
I20260928 14:57:03.861092 6923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:03.834227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:42423 (local address 127.31.40.61:37667)
0928 14:57:03.834693 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:03.834699 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:03.835093 (+ 394us) server_negotiation.cc:407] Connection header received
0928 14:57:03.836058 (+ 965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:03.836063 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:03.836145 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:03.836253 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:03.839720 (+ 3467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:03.842879 (+ 3159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:03.845872 (+ 2993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:03.846364 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:03.846500 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:03.851841 (+ 5341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:03.851851 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:03.851858 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:03.851931 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:03.856398 (+ 4467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:03.857665 (+ 1267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:03.857674 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:03.857680 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:03.857870 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:03.858456 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:03.858481 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:03.858487 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:03.858994 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:03.859125 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:03.860329 (+ 1204us) server_negotiation.cc:299] Negotiation successful
0928 14:57:03.860602 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":229,"threads_started":1}
I20260928 14:57:03.862362 6919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:03.830134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:51055 (local address 127.31.40.60:42157)
0928 14:57:03.831849 (+ 1715us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:03.831858 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:03.833343 (+ 1485us) server_negotiation.cc:407] Connection header received
0928 14:57:03.835982 (+ 2639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:03.835988 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:03.836080 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:03.836163 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:03.838449 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:03.839200 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:03.841613 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:03.842074 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:03.842222 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:03.848322 (+ 6100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:03.848329 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:03.848333 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:03.848374 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:03.852839 (+ 4465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:03.859061 (+ 6222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:03.859067 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:03.859070 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:03.859257 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:03.860051 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:03.860056 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:03.860059 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:03.860414 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:03.860551 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:03.861718 (+ 1167us) server_negotiation.cc:299] Negotiation successful
0928 14:57:03.862020 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1567,"thread_start_us":1475,"threads_started":1}
I20260928 14:57:03.863686 6922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:03.834010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.61:37667 (local address 127.31.40.3:42423)
0928 14:57:03.834974 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:03.834994 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:03.835632 (+ 638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:03.836579 (+ 947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:03.836589 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:03.839006 (+ 2417us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:03.839465 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:03.839485 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:03.843227 (+ 3742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:03.843235 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:03.845701 (+ 2466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:03.845715 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:03.846585 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:03.846591 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:03.846738 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:03.847920 (+ 1182us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:03.847946 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:03.851584 (+ 3638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:03.856682 (+ 5098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:03.856690 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:03.856697 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:03.857393 (+ 696us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:03.858025 (+ 632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:03.858029 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:03.858032 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:03.858300 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:03.859245 (+ 945us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:03.859253 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:03.859812 (+ 559us) client_negotiation.cc:769] Sending connection context
0928 14:57:03.861496 (+ 1684us) client_negotiation.cc:240] Negotiation successful
0928 14:57:03.862131 (+ 635us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":841,"thread_start_us":671,"threads_started":1}
I20260928 14:57:03.864399 6920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:03.831089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:42157 (local address 127.31.40.3:51055)
0928 14:57:03.833179 (+ 2090us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:03.833219 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:03.835637 (+ 2418us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:03.836577 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:03.836590 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:03.837505 (+ 915us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:03.838240 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:03.838264 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:03.839383 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:03.839394 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:03.841446 (+ 2052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:03.841458 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:03.842327 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:03.842335 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:03.842648 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:03.843532 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:03.843571 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:03.848012 (+ 4441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:03.857849 (+ 9837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:03.857855 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:03.857858 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:03.858323 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:03.859438 (+ 1115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:03.859442 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:03.859445 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:03.859606 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:03.860668 (+ 1062us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:03.860673 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:03.860831 (+ 158us) client_negotiation.cc:769] Sending connection context
0928 14:57:03.862863 (+ 2032us) client_negotiation.cc:240] Negotiation successful
0928 14:57:03.863165 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1711,"spinlock_wait_cycles":94080,"thread_start_us":2278,"threads_started":1}
I20260928 14:57:03.865837 6916 heartbeater.cc:347] Connected to a master server at 127.31.40.61:37667
I20260928 14:57:03.865959 6914 heartbeater.cc:347] Connected to a master server at 127.31.40.60:42157
I20260928 14:57:03.866305 6914 heartbeater.cc:464] Registering TS with master...
I20260928 14:57:03.867530 6914 heartbeater.cc:511] Master 127.31.40.60:42157 requested a full tablet report, sending...
I20260928 14:57:03.866322 6916 heartbeater.cc:464] Registering TS with master...
I20260928 14:57:03.869223 6916 heartbeater.cc:511] Master 127.31.40.61:37667 requested a full tablet report, sending...
I20260928 14:57:03.871191 6256 ts_manager.cc:194] Registered new tserver with Master: f2f0f28f0299472386abafe15704df7b (127.31.40.3:34255)
I20260928 14:57:03.872641 6445 ts_manager.cc:194] Registered new tserver with Master: f2f0f28f0299472386abafe15704df7b (127.31.40.3:34255)
I20260928 14:57:03.873886 6921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:03.831854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:32909 (local address 127.31.40.3:32969)
0928 14:57:03.833180 (+ 1326us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:03.833220 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:03.835632 (+ 2412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:03.837081 (+ 1449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:03.837086 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:03.837718 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:03.838305 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:03.838326 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:03.840757 (+ 2431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:03.840764 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:03.842989 (+ 2225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:03.843011 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:03.844140 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:03.844150 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:03.844358 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:03.845030 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:03.845060 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:03.848012 (+ 2952us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:03.855653 (+ 7641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:03.855669 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:03.855699 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:03.856730 (+ 1031us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:03.858533 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:03.858540 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:03.858547 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:03.858751 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:03.872249 (+ 13498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:03.872258 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:03.872557 (+ 299us) client_negotiation.cc:769] Sending connection context
0928 14:57:03.872760 (+ 203us) client_negotiation.cc:240] Negotiation successful
0928 14:57:03.873213 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1068,"spinlock_wait_cycles":11264,"thread_start_us":688,"threads_started":1}
I20260928 14:57:03.877094 6918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:03.830109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:32969 (local address 127.31.40.62:32909)
0928 14:57:03.830659 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:03.830757 (+ 98us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:03.834926 (+ 4169us) server_negotiation.cc:407] Connection header received
0928 14:57:03.836820 (+ 1894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:03.836828 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:03.836922 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:03.837056 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:03.839429 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:03.840590 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:03.843177 (+ 2587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:03.843899 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:03.844051 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:03.848294 (+ 4243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:03.848303 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:03.848307 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:03.848352 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:03.855382 (+ 7030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:03.856984 (+ 1602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:03.857005 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:03.857012 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:03.857223 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:03.863756 (+ 6533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:03.863766 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:03.863772 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:03.864303 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:03.875938 (+ 11635us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:03.876161 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 14:57:03.876528 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":249,"threads_started":1}
I20260928 14:57:03.878341 6915 heartbeater.cc:347] Connected to a master server at 127.31.40.62:32909
I20260928 14:57:03.878510 6915 heartbeater.cc:464] Registering TS with master...
I20260928 14:57:03.879045 6915 heartbeater.cc:511] Master 127.31.40.62:32909 requested a full tablet report, sending...
I20260928 14:57:03.880262 5655 ts_manager.cc:194] Re-registered known tserver with Master: f2f0f28f0299472386abafe15704df7b (127.31.40.3:34255)
I20260928 14:57:03.881752 5655 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.40.3@KRBTEST.COM'} at 127.31.40.3:32969
I20260928 14:57:03.899034 6918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:03.849289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59412 (local address 127.31.40.62:32909)
0928 14:57:03.877421 (+ 28132us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:03.877430 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:03.877454 (+ 24us) server_negotiation.cc:407] Connection header received
0928 14:57:03.877539 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:03.877547 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:03.877647 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:03.877778 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:03.883880 (+ 6102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:03.885057 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:03.886909 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:03.887556 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:03.887716 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:03.891378 (+ 3662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:03.891392 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:03.891397 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:03.891431 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:03.894810 (+ 3379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:03.895789 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:03.895796 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:03.895800 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:03.895981 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:03.896454 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:03.896461 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:03.896467 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:03.896886 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:03.897168 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:03.898355 (+ 1187us) server_negotiation.cc:299] Negotiation successful
0928 14:57:03.898624 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27986,"thread_start_us":29729,"threads_started":1}
I20260928 14:57:03.902156 6923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:03.850865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58358 (local address 127.31.40.61:37667)
0928 14:57:03.861408 (+ 10543us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:03.861414 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:03.884534 (+ 23120us) server_negotiation.cc:407] Connection header received
0928 14:57:03.885082 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:03.885087 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:03.885193 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:03.885304 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:03.888324 (+ 3020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:03.889409 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:03.890953 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:03.891377 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:03.891524 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:03.894506 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:03.894513 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:03.894518 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:03.894564 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:03.898006 (+ 3442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:03.898840 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:03.898848 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:03.898854 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:03.899013 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:03.899484 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:03.899492 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:03.899498 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:03.899915 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:03.900168 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:03.901364 (+ 1196us) server_negotiation.cc:299] Negotiation successful
0928 14:57:03.901649 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10441,"thread_start_us":33803,"threads_started":1}
I20260928 14:57:03.905032 6919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:03.878753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45176 (local address 127.31.40.60:42157)
0928 14:57:03.878923 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:03.878931 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:03.887414 (+ 8483us) server_negotiation.cc:407] Connection header received
0928 14:57:03.887565 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:03.887571 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:03.887658 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:03.887759 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:03.889998 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:03.891129 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:03.893722 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:03.894371 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:03.894493 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:03.897508 (+ 3015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:03.897516 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:03.897522 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:03.897568 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:03.901236 (+ 3668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:03.901912 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:03.901918 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:03.901922 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:03.902061 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:03.902562 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:03.902568 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:03.902571 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:03.902875 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:03.903182 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:03.904444 (+ 1262us) server_negotiation.cc:299] Negotiation successful
0928 14:57:03.904708 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 14:57:03.905907 5655 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59412:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:57:03.913700 5655 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:57:03.948587 6945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:03.935536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46372 (local address 127.31.40.1:35061)
0928 14:57:03.936219 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:03.936234 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:03.937415 (+ 1181us) server_negotiation.cc:407] Connection header received
0928 14:57:03.937794 (+ 379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:03.937807 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:03.938183 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:03.938384 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:03.939441 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:03.941598 (+ 2157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:03.944689 (+ 3091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:03.946476 (+ 1787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:03.946583 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:03.946832 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:03.948046 (+ 1214us) server_negotiation.cc:299] Negotiation successful
0928 14:57:03.948287 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":327,"threads_started":1}
I20260928 14:57:03.948616 6946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:03.935320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:35061 (local address 127.0.0.1:46372)
0928 14:57:03.937292 (+ 1972us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:03.937313 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:03.937549 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:03.938714 (+ 1165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:03.938721 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:03.938754 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:03.939215 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:03.939230 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:03.941766 (+ 2536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:03.941775 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:03.944509 (+ 2734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:03.944524 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:03.947272 (+ 2748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:03.947278 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:03.947755 (+ 477us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:03.947796 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 14:57:03.947929 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 14:57:03.948105 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1822,"mutex_wait_us":215,"thread_start_us":1987,"threads_started":1}
I20260928 14:57:03.951289 6949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:03.937418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32802 (local address 127.31.40.3:34255)
0928 14:57:03.941183 (+ 3765us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:03.941196 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:03.941240 (+ 44us) server_negotiation.cc:407] Connection header received
0928 14:57:03.941367 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:03.941380 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:03.942829 (+ 1449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:03.943043 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:03.943999 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:03.945460 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:03.948497 (+ 3037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:03.950128 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:03.950255 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:03.950481 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:03.950783 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 14:57:03.950961 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3562,"thread_start_us":3401,"threads_started":1}
I20260928 14:57:03.951836 6944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:03.936686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.3:34255 (local address 127.0.0.1:32802)
0928 14:57:03.937581 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:03.937597 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:03.937721 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:03.943023 (+ 5302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:03.943028 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:03.943050 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:03.943504 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:03.943517 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:03.945622 (+ 2105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:03.945628 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:03.948249 (+ 2621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:03.948269 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:03.950342 (+ 2073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:03.950349 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:03.950543 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:03.950576 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 14:57:03.950738 (+ 162us) client_negotiation.cc:240] Negotiation successful
0928 14:57:03.950865 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753,"mutex_wait_us":54,"thread_start_us":325,"threads_started":1}
I20260928 14:57:03.953394 6559 tablet_service.cc:1467] Processing CreateTablet for tablet 75988fcc410a4a29a39e01f26a355f62 (DEFAULT_TABLE table=test-table [id=ce209d17e4e942a98a6ce0eeaeca1f3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:57:03.954290 6947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:03.936978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.2:33025 (local address 127.0.0.1:42946)
0928 14:57:03.942658 (+ 5680us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:03.942678 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:03.942822 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:03.943457 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:03.943465 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:03.943497 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:03.944060 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:03.944074 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:03.947787 (+ 3713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:03.947792 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:03.950395 (+ 2603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:03.950409 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:03.953481 (+ 3072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:03.953496 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:03.953682 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:03.953720 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 14:57:03.953828 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 14:57:03.953960 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5549,"thread_start_us":5235,"threads_started":1}
I20260928 14:57:03.954342 6948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:03.937138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42946 (local address 127.31.40.2:33025)
0928 14:57:03.942680 (+ 5542us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:03.942688 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:03.942794 (+ 106us) server_negotiation.cc:407] Connection header received
0928 14:57:03.942982 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:03.942990 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:03.943249 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:03.943467 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:03.944261 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:03.947675 (+ 3414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:03.951968 (+ 4293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:03.953271 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:03.953427 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:03.953622 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:03.953898 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 14:57:03.954078 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5385,"thread_start_us":5287,"threads_started":1}
I20260928 14:57:03.955366 6559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75988fcc410a4a29a39e01f26a355f62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:03.955896 6849 tablet_service.cc:1467] Processing CreateTablet for tablet 75988fcc410a4a29a39e01f26a355f62 (DEFAULT_TABLE table=test-table [id=ce209d17e4e942a98a6ce0eeaeca1f3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:57:03.957250 6849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75988fcc410a4a29a39e01f26a355f62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:03.957520 6708 tablet_service.cc:1467] Processing CreateTablet for tablet 75988fcc410a4a29a39e01f26a355f62 (DEFAULT_TABLE table=test-table [id=ce209d17e4e942a98a6ce0eeaeca1f3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:57:03.958612 6708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75988fcc410a4a29a39e01f26a355f62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:03.972882 6950 tablet_bootstrap.cc:492] T 75988fcc410a4a29a39e01f26a355f62 P 50bbf97992ac4d92a0ce12b7b1be3bbe: Bootstrap starting.
I20260928 14:57:03.974306 6951 tablet_bootstrap.cc:492] T 75988fcc410a4a29a39e01f26a355f62 P 1484aa0f119348a1af94cea1a988d0f2: Bootstrap starting.
I20260928 14:57:03.975713 6952 tablet_bootstrap.cc:492] T 75988fcc410a4a29a39e01f26a355f62 P f2f0f28f0299472386abafe15704df7b: Bootstrap starting.
I20260928 14:57:03.978209 6951 tablet_bootstrap.cc:654] T 75988fcc410a4a29a39e01f26a355f62 P 1484aa0f119348a1af94cea1a988d0f2: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:03.978804 6952 tablet_bootstrap.cc:654] T 75988fcc410a4a29a39e01f26a355f62 P f2f0f28f0299472386abafe15704df7b: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:03.980126 6952 log.cc:824] T 75988fcc410a4a29a39e01f26a355f62 P f2f0f28f0299472386abafe15704df7b: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:03.982730 6950 tablet_bootstrap.cc:654] T 75988fcc410a4a29a39e01f26a355f62 P 50bbf97992ac4d92a0ce12b7b1be3bbe: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:03.984807 6951 log.cc:824] T 75988fcc410a4a29a39e01f26a355f62 P 1484aa0f119348a1af94cea1a988d0f2: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:03.984999 6952 tablet_bootstrap.cc:492] T 75988fcc410a4a29a39e01f26a355f62 P f2f0f28f0299472386abafe15704df7b: No bootstrap required, opened a new log
I20260928 14:57:03.985009 6950 log.cc:824] T 75988fcc410a4a29a39e01f26a355f62 P 50bbf97992ac4d92a0ce12b7b1be3bbe: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:03.985350 6952 ts_tablet_manager.cc:1397] T 75988fcc410a4a29a39e01f26a355f62 P f2f0f28f0299472386abafe15704df7b: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260928 14:57:03.991956 6950 tablet_bootstrap.cc:492] T 75988fcc410a4a29a39e01f26a355f62 P 50bbf97992ac4d92a0ce12b7b1be3bbe: No bootstrap required, opened a new log
I20260928 14:57:03.992411 6950 ts_tablet_manager.cc:1397] T 75988fcc410a4a29a39e01f26a355f62 P 50bbf97992ac4d92a0ce12b7b1be3bbe: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.003s
I20260928 14:57:04.002071 6951 tablet_bootstrap.cc:492] T 75988fcc410a4a29a39e01f26a355f62 P 1484aa0f119348a1af94cea1a988d0f2: No bootstrap required, opened a new log
I20260928 14:57:04.001946 6950 raft_consensus.cc:348] T 75988fcc410a4a29a39e01f26a355f62 P 50bbf97992ac4d92a0ce12b7b1be3bbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50bbf97992ac4d92a0ce12b7b1be3bbe" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 35061 } } peers { permanent_uuid: "f2f0f28f0299472386abafe15704df7b" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34255 } } peers { permanent_uuid: "1484aa0f119348a1af94cea1a988d0f2" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 33025 } }
I20260928 14:57:04.001948 6952 raft_consensus.cc:348] T 75988fcc410a4a29a39e01f26a355f62 P f2f0f28f0299472386abafe15704df7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50bbf97992ac4d92a0ce12b7b1be3bbe" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 35061 } } peers { permanent_uuid: "f2f0f28f0299472386abafe15704df7b" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34255 } } peers { permanent_uuid: "1484aa0f119348a1af94cea1a988d0f2" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 33025 } }
I20260928 14:57:04.002403 6951 ts_tablet_manager.cc:1397] T 75988fcc410a4a29a39e01f26a355f62 P 1484aa0f119348a1af94cea1a988d0f2: Time spent bootstrapping tablet: real 0.029s user 0.010s sys 0.010s
I20260928 14:57:04.002440 6952 raft_consensus.cc:374] T 75988fcc410a4a29a39e01f26a355f62 P f2f0f28f0299472386abafe15704df7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:04.002437 6950 raft_consensus.cc:374] T 75988fcc410a4a29a39e01f26a355f62 P 50bbf97992ac4d92a0ce12b7b1be3bbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:04.002540 6950 raft_consensus.cc:729] T 75988fcc410a4a29a39e01f26a355f62 P 50bbf97992ac4d92a0ce12b7b1be3bbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50bbf97992ac4d92a0ce12b7b1be3bbe, State: Initialized, Role: FOLLOWER
I20260928 14:57:04.002552 6952 raft_consensus.cc:729] T 75988fcc410a4a29a39e01f26a355f62 P f2f0f28f0299472386abafe15704df7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2f0f28f0299472386abafe15704df7b, State: Initialized, Role: FOLLOWER
I20260928 14:57:04.003307 6950 consensus_queue.cc:260] T 75988fcc410a4a29a39e01f26a355f62 P 50bbf97992ac4d92a0ce12b7b1be3bbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50bbf97992ac4d92a0ce12b7b1be3bbe" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 35061 } } peers { permanent_uuid: "f2f0f28f0299472386abafe15704df7b" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34255 } } peers { permanent_uuid: "1484aa0f119348a1af94cea1a988d0f2" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 33025 } }
I20260928 14:57:04.003328 6952 consensus_queue.cc:260] T 75988fcc410a4a29a39e01f26a355f62 P f2f0f28f0299472386abafe15704df7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50bbf97992ac4d92a0ce12b7b1be3bbe" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 35061 } } peers { permanent_uuid: "f2f0f28f0299472386abafe15704df7b" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34255 } } peers { permanent_uuid: "1484aa0f119348a1af94cea1a988d0f2" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 33025 } }
I20260928 14:57:04.009708 6950 ts_tablet_manager.cc:1428] T 75988fcc410a4a29a39e01f26a355f62 P 50bbf97992ac4d92a0ce12b7b1be3bbe: Time spent starting tablet: real 0.017s user 0.005s sys 0.005s
I20260928 14:57:04.016129 6952 ts_tablet_manager.cc:1428] T 75988fcc410a4a29a39e01f26a355f62 P f2f0f28f0299472386abafe15704df7b: Time spent starting tablet: real 0.031s user 0.024s sys 0.004s
I20260928 14:57:04.016585 6915 heartbeater.cc:503] Master 127.31.40.62:32909 was elected leader, sending a full tablet report...
I20260928 14:57:04.017704 6951 raft_consensus.cc:348] T 75988fcc410a4a29a39e01f26a355f62 P 1484aa0f119348a1af94cea1a988d0f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50bbf97992ac4d92a0ce12b7b1be3bbe" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 35061 } } peers { permanent_uuid: "f2f0f28f0299472386abafe15704df7b" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34255 } } peers { permanent_uuid: "1484aa0f119348a1af94cea1a988d0f2" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 33025 } }
I20260928 14:57:04.018378 6951 raft_consensus.cc:374] T 75988fcc410a4a29a39e01f26a355f62 P 1484aa0f119348a1af94cea1a988d0f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:04.018532 6951 raft_consensus.cc:729] T 75988fcc410a4a29a39e01f26a355f62 P 1484aa0f119348a1af94cea1a988d0f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1484aa0f119348a1af94cea1a988d0f2, State: Initialized, Role: FOLLOWER
I20260928 14:57:04.019309 6951 consensus_queue.cc:260] T 75988fcc410a4a29a39e01f26a355f62 P 1484aa0f119348a1af94cea1a988d0f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50bbf97992ac4d92a0ce12b7b1be3bbe" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 35061 } } peers { permanent_uuid: "f2f0f28f0299472386abafe15704df7b" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34255 } } peers { permanent_uuid: "1484aa0f119348a1af94cea1a988d0f2" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 33025 } }
I20260928 14:57:04.021538 6774 heartbeater.cc:503] Master 127.31.40.62:32909 was elected leader, sending a full tablet report...
I20260928 14:57:04.022411 6951 ts_tablet_manager.cc:1428] T 75988fcc410a4a29a39e01f26a355f62 P 1484aa0f119348a1af94cea1a988d0f2: Time spent starting tablet: real 0.020s user 0.017s sys 0.002s
W20260928 14:57:04.083220 6917 tablet.cc:2369] T 75988fcc410a4a29a39e01f26a355f62 P f2f0f28f0299472386abafe15704df7b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:57:04.118700 6956 raft_consensus.cc:482] T 75988fcc410a4a29a39e01f26a355f62 P 50bbf97992ac4d92a0ce12b7b1be3bbe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:57:04.118986 6956 raft_consensus.cc:504] T 75988fcc410a4a29a39e01f26a355f62 P 50bbf97992ac4d92a0ce12b7b1be3bbe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50bbf97992ac4d92a0ce12b7b1be3bbe" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 35061 } } peers { permanent_uuid: "f2f0f28f0299472386abafe15704df7b" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34255 } } peers { permanent_uuid: "1484aa0f119348a1af94cea1a988d0f2" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 33025 } }
I20260928 14:57:04.120867 6956 leader_election.cc:288] T 75988fcc410a4a29a39e01f26a355f62 P 50bbf97992ac4d92a0ce12b7b1be3bbe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2f0f28f0299472386abafe15704df7b (127.31.40.3:34255), 1484aa0f119348a1af94cea1a988d0f2 (127.31.40.2:33025)
I20260928 14:57:04.128729 6949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:04.121300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:39809 (local address 127.31.40.3:34255)
0928 14:57:04.121497 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:04.121502 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:04.122388 (+ 886us) server_negotiation.cc:407] Connection header received
0928 14:57:04.122624 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:04.122631 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:04.122699 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:04.122802 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:04.123501 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:04.124328 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:04.126775 (+ 2447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:04.127639 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:04.127824 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:04.128010 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:04.128303 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 14:57:04.128431 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 14:57:04.128732 6948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:04.121292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:55559 (local address 127.31.40.2:33025)
0928 14:57:04.121492 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:04.121498 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:04.122941 (+ 1443us) server_negotiation.cc:407] Connection header received
0928 14:57:04.123212 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:04.123216 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:04.123274 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:04.123372 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:04.123912 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:04.124715 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:04.126741 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:04.127614 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:04.127797 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:04.127929 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:04.128242 (+ 313us) server_negotiation.cc:299] Negotiation successful
0928 14:57:04.128369 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 14:57:04.128916 6959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:04.121149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.3:34255 (local address 127.31.40.1:39809)
0928 14:57:04.122265 (+ 1116us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:04.122282 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:04.122472 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:04.122821 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:04.122825 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:04.122849 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:04.123343 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:04.123354 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:04.124562 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:04.124569 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:04.126624 (+ 2055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:04.126637 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:04.127995 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:04.128007 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:04.128145 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:04.128167 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 14:57:04.128276 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 14:57:04.128405 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":998,"mutex_wait_us":32,"thread_start_us":749,"threads_started":1}
I20260928 14:57:04.128931 6960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:04.121309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.2:33025 (local address 127.31.40.1:55559)
0928 14:57:04.122824 (+ 1515us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:04.122839 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:04.123061 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:04.123385 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:04.123390 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:04.123411 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:04.123721 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:04.123730 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:04.124863 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:04.124868 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:04.126575 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:04.126588 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:04.127898 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:04.127905 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:04.128080 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:04.128104 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 14:57:04.128215 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 14:57:04.128351 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1385,"mutex_wait_us":21,"thread_start_us":1188,"threads_started":1}
I20260928 14:57:04.130259 6728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75988fcc410a4a29a39e01f26a355f62" candidate_uuid: "50bbf97992ac4d92a0ce12b7b1be3bbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1484aa0f119348a1af94cea1a988d0f2" is_pre_election: true
I20260928 14:57:04.130259 6869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75988fcc410a4a29a39e01f26a355f62" candidate_uuid: "50bbf97992ac4d92a0ce12b7b1be3bbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2f0f28f0299472386abafe15704df7b" is_pre_election: true
I20260928 14:57:04.130911 6869 raft_consensus.cc:2454] T 75988fcc410a4a29a39e01f26a355f62 P f2f0f28f0299472386abafe15704df7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50bbf97992ac4d92a0ce12b7b1be3bbe in term 0.
I20260928 14:57:04.130911 6728 raft_consensus.cc:2454] T 75988fcc410a4a29a39e01f26a355f62 P 1484aa0f119348a1af94cea1a988d0f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50bbf97992ac4d92a0ce12b7b1be3bbe in term 0.
I20260928 14:57:04.131682 6514 leader_election.cc:302] T 75988fcc410a4a29a39e01f26a355f62 P 50bbf97992ac4d92a0ce12b7b1be3bbe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1484aa0f119348a1af94cea1a988d0f2, 50bbf97992ac4d92a0ce12b7b1be3bbe, f2f0f28f0299472386abafe15704df7b; no voters:
I20260928 14:57:04.132205 6956 raft_consensus.cc:2781] T 75988fcc410a4a29a39e01f26a355f62 P 50bbf97992ac4d92a0ce12b7b1be3bbe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:57:04.132413 6956 raft_consensus.cc:482] T 75988fcc410a4a29a39e01f26a355f62 P 50bbf97992ac4d92a0ce12b7b1be3bbe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:57:04.132550 6956 raft_consensus.cc:3037] T 75988fcc410a4a29a39e01f26a355f62 P 50bbf97992ac4d92a0ce12b7b1be3bbe [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:04.134963 6956 raft_consensus.cc:504] T 75988fcc410a4a29a39e01f26a355f62 P 50bbf97992ac4d92a0ce12b7b1be3bbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50bbf97992ac4d92a0ce12b7b1be3bbe" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 35061 } } peers { permanent_uuid: "f2f0f28f0299472386abafe15704df7b" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34255 } } peers { permanent_uuid: "1484aa0f119348a1af94cea1a988d0f2" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 33025 } }
I20260928 14:57:04.135612 6956 leader_election.cc:288] T 75988fcc410a4a29a39e01f26a355f62 P 50bbf97992ac4d92a0ce12b7b1be3bbe [CANDIDATE]: Term 1 election: Requested vote from peers f2f0f28f0299472386abafe15704df7b (127.31.40.3:34255), 1484aa0f119348a1af94cea1a988d0f2 (127.31.40.2:33025)
I20260928 14:57:04.136327 6728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75988fcc410a4a29a39e01f26a355f62" candidate_uuid: "50bbf97992ac4d92a0ce12b7b1be3bbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1484aa0f119348a1af94cea1a988d0f2"
I20260928 14:57:04.136590 6728 raft_consensus.cc:3037] T 75988fcc410a4a29a39e01f26a355f62 P 1484aa0f119348a1af94cea1a988d0f2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:04.136586 6869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75988fcc410a4a29a39e01f26a355f62" candidate_uuid: "50bbf97992ac4d92a0ce12b7b1be3bbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2f0f28f0299472386abafe15704df7b"
I20260928 14:57:04.136868 6869 raft_consensus.cc:3037] T 75988fcc410a4a29a39e01f26a355f62 P f2f0f28f0299472386abafe15704df7b [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:04.139956 6869 raft_consensus.cc:2454] T 75988fcc410a4a29a39e01f26a355f62 P f2f0f28f0299472386abafe15704df7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50bbf97992ac4d92a0ce12b7b1be3bbe in term 1.
I20260928 14:57:04.140208 6728 raft_consensus.cc:2454] T 75988fcc410a4a29a39e01f26a355f62 P 1484aa0f119348a1af94cea1a988d0f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50bbf97992ac4d92a0ce12b7b1be3bbe in term 1.
I20260928 14:57:04.140491 6514 leader_election.cc:302] T 75988fcc410a4a29a39e01f26a355f62 P 50bbf97992ac4d92a0ce12b7b1be3bbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50bbf97992ac4d92a0ce12b7b1be3bbe, f2f0f28f0299472386abafe15704df7b; no voters:
I20260928 14:57:04.140959 6956 raft_consensus.cc:2781] T 75988fcc410a4a29a39e01f26a355f62 P 50bbf97992ac4d92a0ce12b7b1be3bbe [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:57:04.141924 6956 raft_consensus.cc:686] T 75988fcc410a4a29a39e01f26a355f62 P 50bbf97992ac4d92a0ce12b7b1be3bbe [term 1 LEADER]: Becoming Leader. State: Replica: 50bbf97992ac4d92a0ce12b7b1be3bbe, State: Running, Role: LEADER
I20260928 14:57:04.142618 6956 consensus_queue.cc:237] T 75988fcc410a4a29a39e01f26a355f62 P 50bbf97992ac4d92a0ce12b7b1be3bbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50bbf97992ac4d92a0ce12b7b1be3bbe" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 35061 } } peers { permanent_uuid: "f2f0f28f0299472386abafe15704df7b" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34255 } } peers { permanent_uuid: "1484aa0f119348a1af94cea1a988d0f2" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 33025 } }
I20260928 14:57:04.149515 5654 catalog_manager.cc:5495] T 75988fcc410a4a29a39e01f26a355f62 P 50bbf97992ac4d92a0ce12b7b1be3bbe reported cstate change: term changed from 0 to 1, leader changed from <none> to 50bbf97992ac4d92a0ce12b7b1be3bbe (127.31.40.1). New cstate: current_term: 1 leader_uuid: "50bbf97992ac4d92a0ce12b7b1be3bbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50bbf97992ac4d92a0ce12b7b1be3bbe" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 35061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f2f0f28f0299472386abafe15704df7b" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1484aa0f119348a1af94cea1a988d0f2" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 33025 } health_report { overall_health: UNKNOWN } } }
W20260928 14:57:04.184404 6776 tablet.cc:2369] T 75988fcc410a4a29a39e01f26a355f62 P 1484aa0f119348a1af94cea1a988d0f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:57:04.197084 6945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:04.191810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46388 (local address 127.31.40.1:35061)
0928 14:57:04.192024 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:04.192029 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:04.192050 (+ 21us) server_negotiation.cc:407] Connection header received
0928 14:57:04.192294 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:04.192298 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:04.192373 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:04.192520 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:57:04.192885 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:04.193645 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:04.194942 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:04.195341 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:04.195478 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:04.196011 (+ 533us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:57:04.196522 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:57:04.196691 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:04.196740 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 14:57:04.196820 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 14:57:04.205446 6728 raft_consensus.cc:1261] T 75988fcc410a4a29a39e01f26a355f62 P 1484aa0f119348a1af94cea1a988d0f2 [term 1 FOLLOWER]: Refusing update from remote peer 50bbf97992ac4d92a0ce12b7b1be3bbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:57:04.205540 6869 raft_consensus.cc:1261] T 75988fcc410a4a29a39e01f26a355f62 P f2f0f28f0299472386abafe15704df7b [term 1 FOLLOWER]: Refusing update from remote peer 50bbf97992ac4d92a0ce12b7b1be3bbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:57:04.206560 6961 consensus_queue.cc:1035] T 75988fcc410a4a29a39e01f26a355f62 P 50bbf97992ac4d92a0ce12b7b1be3bbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2f0f28f0299472386abafe15704df7b" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:57:04.206878 6956 consensus_queue.cc:1035] T 75988fcc410a4a29a39e01f26a355f62 P 50bbf97992ac4d92a0ce12b7b1be3bbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "1484aa0f119348a1af94cea1a988d0f2" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 33025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:57:04.221179 6965 mvcc.cc:205] Tried to move back new op lower bound from 7334328009535283200 to 7334328009292918784. Current Snapshot: MvccSnapshot[applied={T|T < 7334328009535283200}]
I20260928 14:57:04.236424 5654 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:57:04.236766 5654 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:57:04.245421 5654 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 40f51af6fc714868a4c4df423f897b6c: Sending DeleteTablet for 3 replicas of tablet 75988fcc410a4a29a39e01f26a355f62
I20260928 14:57:04.246593 6559 tablet_service.cc:1514] Processing DeleteTablet for tablet 75988fcc410a4a29a39e01f26a355f62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:57:04 UTC) from {username='oryx', principal='oryx/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:46372
I20260928 14:57:04.246735 6708 tablet_service.cc:1514] Processing DeleteTablet for tablet 75988fcc410a4a29a39e01f26a355f62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:57:04 UTC) from {username='oryx', principal='oryx/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:42946
I20260928 14:57:04.246904 6849 tablet_service.cc:1514] Processing DeleteTablet for tablet 75988fcc410a4a29a39e01f26a355f62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:57:04 UTC) from {username='oryx', principal='oryx/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:32802
I20260928 14:57:04.259975 6974 tablet_replica.cc:330] T 75988fcc410a4a29a39e01f26a355f62 P 50bbf97992ac4d92a0ce12b7b1be3bbe: stopping tablet replica
I20260928 14:57:04.260424 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 5623
I20260928 14:57:04.261175 6974 raft_consensus.cc:2229] T 75988fcc410a4a29a39e01f26a355f62 P 50bbf97992ac4d92a0ce12b7b1be3bbe [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:57:04.261602 5685 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:57:04.261997 6974 raft_consensus.cc:2258] T 75988fcc410a4a29a39e01f26a355f62 P 50bbf97992ac4d92a0ce12b7b1be3bbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:57:04.269002 6976 tablet_replica.cc:330] T 75988fcc410a4a29a39e01f26a355f62 P f2f0f28f0299472386abafe15704df7b: stopping tablet replica
I20260928 14:57:04.269673 6976 raft_consensus.cc:2229] T 75988fcc410a4a29a39e01f26a355f62 P f2f0f28f0299472386abafe15704df7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:57:04.269897 6975 tablet_replica.cc:330] T 75988fcc410a4a29a39e01f26a355f62 P 1484aa0f119348a1af94cea1a988d0f2: stopping tablet replica
I20260928 14:57:04.270284 6974 ts_tablet_manager.cc:1905] T 75988fcc410a4a29a39e01f26a355f62 P 50bbf97992ac4d92a0ce12b7b1be3bbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:57:04.270414 6975 raft_consensus.cc:2229] T 75988fcc410a4a29a39e01f26a355f62 P 1484aa0f119348a1af94cea1a988d0f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:57:04.271494 6975 raft_consensus.cc:2258] T 75988fcc410a4a29a39e01f26a355f62 P 1484aa0f119348a1af94cea1a988d0f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:57:04.271725 6976 raft_consensus.cc:2258] T 75988fcc410a4a29a39e01f26a355f62 P f2f0f28f0299472386abafe15704df7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:57:04.274053 6975 ts_tablet_manager.cc:1905] T 75988fcc410a4a29a39e01f26a355f62 P 1484aa0f119348a1af94cea1a988d0f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:57:04.277446 6976 ts_tablet_manager.cc:1905] T 75988fcc410a4a29a39e01f26a355f62 P f2f0f28f0299472386abafe15704df7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:57:04.279423 6974 ts_tablet_manager.cc:1918] T 75988fcc410a4a29a39e01f26a355f62 P 50bbf97992ac4d92a0ce12b7b1be3bbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:57:04.285812 6976 ts_tablet_manager.cc:1918] T 75988fcc410a4a29a39e01f26a355f62 P f2f0f28f0299472386abafe15704df7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:57:04.286093 6976 log.cc:1196] T 75988fcc410a4a29a39e01f26a355f62 P f2f0f28f0299472386abafe15704df7b: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-2/wal/wals/75988fcc410a4a29a39e01f26a355f62
I20260928 14:57:04.286679 6976 ts_tablet_manager.cc:1939] T 75988fcc410a4a29a39e01f26a355f62 P f2f0f28f0299472386abafe15704df7b: Deleting consensus metadata
I20260928 14:57:04.291220 6975 ts_tablet_manager.cc:1918] T 75988fcc410a4a29a39e01f26a355f62 P 1484aa0f119348a1af94cea1a988d0f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:57:04.291538 6975 log.cc:1196] T 75988fcc410a4a29a39e01f26a355f62 P 1484aa0f119348a1af94cea1a988d0f2: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-1/wal/wals/75988fcc410a4a29a39e01f26a355f62
I20260928 14:57:04.292210 6975 ts_tablet_manager.cc:1939] T 75988fcc410a4a29a39e01f26a355f62 P 1484aa0f119348a1af94cea1a988d0f2: Deleting consensus metadata
I20260928 14:57:04.294560 6974 log.cc:1196] T 75988fcc410a4a29a39e01f26a355f62 P 50bbf97992ac4d92a0ce12b7b1be3bbe: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-0/wal/wals/75988fcc410a4a29a39e01f26a355f62
I20260928 14:57:04.295346 6974 ts_tablet_manager.cc:1939] T 75988fcc410a4a29a39e01f26a355f62 P 50bbf97992ac4d92a0ce12b7b1be3bbe: Deleting consensus metadata
I20260928 14:57:04.298602 6923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:04.249277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58368 (local address 127.31.40.61:37667)
0928 14:57:04.249496 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:04.249505 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:04.259424 (+ 9919us) server_negotiation.cc:407] Connection header received
0928 14:57:04.263053 (+ 3629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:04.263061 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:04.263173 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:04.263268 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:04.266814 (+ 3546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:04.268054 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:04.279735 (+ 11681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:04.280473 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:04.280603 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:04.284700 (+ 4097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:04.284707 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:04.284712 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:04.284764 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:04.288826 (+ 4062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:04.295260 (+ 6434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:04.295267 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:04.295271 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:04.295531 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:04.296192 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:04.296198 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:04.296202 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:04.296696 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:04.296958 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:04.297885 (+ 927us) server_negotiation.cc:299] Negotiation successful
0928 14:57:04.298133 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 14:57:04.319007 6919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:04.259971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45186 (local address 127.31.40.60:42157)
0928 14:57:04.260191 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:04.260197 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:04.298688 (+ 38491us) server_negotiation.cc:407] Connection header received
0928 14:57:04.299351 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:04.299368 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:04.299504 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:04.299591 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:04.302440 (+ 2849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:04.303661 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:04.306111 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:04.306762 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:04.306891 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:04.311125 (+ 4234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:04.311132 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:04.311138 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:04.311185 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:04.315407 (+ 4222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:04.316605 (+ 1198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:04.316612 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:04.316616 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:04.316773 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:04.317268 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:04.317274 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:04.317278 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:04.317674 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:04.317944 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:04.318399 (+ 455us) server_negotiation.cc:299] Negotiation successful
0928 14:57:04.318639 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 14:57:04.397289 5639 catalog_manager.cc:4841] TS 50bbf97992ac4d92a0ce12b7b1be3bbe (127.31.40.1:35061): tablet 75988fcc410a4a29a39e01f26a355f62 (table test-table [id=ce209d17e4e942a98a6ce0eeaeca1f3c]) successfully deleted
I20260928 14:57:04.397375 5641 catalog_manager.cc:4841] TS f2f0f28f0299472386abafe15704df7b (127.31.40.3:34255): tablet 75988fcc410a4a29a39e01f26a355f62 (table test-table [id=ce209d17e4e942a98a6ce0eeaeca1f3c]) successfully deleted
I20260928 14:57:04.397327 5642 catalog_manager.cc:4841] TS 1484aa0f119348a1af94cea1a988d0f2 (127.31.40.2:33025): tablet 75988fcc410a4a29a39e01f26a355f62 (table test-table [id=ce209d17e4e942a98a6ce0eeaeca1f3c]) successfully deleted
I20260928 14:57:04.398259 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 5623
I20260928 14:57:04.465332 6923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:04.446208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58378 (local address 127.31.40.61:37667)
0928 14:57:04.446504 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:04.446515 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:04.447943 (+ 1428us) server_negotiation.cc:407] Connection header received
0928 14:57:04.448043 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:04.448052 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:04.448162 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:04.448303 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:04.450459 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:04.451658 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:04.453692 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:04.454369 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:04.454473 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:04.458244 (+ 3771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:04.458252 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:04.458258 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:04.458304 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:04.461765 (+ 3461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:04.462679 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:04.462688 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:04.462692 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:04.462845 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:04.463334 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:04.463345 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:04.463352 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:04.463774 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:04.464059 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:04.464684 (+ 625us) server_negotiation.cc:299] Negotiation successful
0928 14:57:04.464944 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 14:57:04.468487 6919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:04.447587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45190 (local address 127.31.40.60:42157)
0928 14:57:04.447860 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:04.447888 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:04.449360 (+ 1472us) server_negotiation.cc:407] Connection header received
0928 14:57:04.449619 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:04.449625 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:04.449702 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:04.449816 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:04.451829 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:04.453067 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:04.454936 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:04.455674 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:04.455781 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:04.461350 (+ 5569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:04.461360 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:04.461368 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:04.461414 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:04.465012 (+ 3598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:04.465869 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:04.465875 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:04.465878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:04.466027 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:04.466612 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:04.466617 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:04.466620 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:04.466996 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:04.467216 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:04.467851 (+ 635us) server_negotiation.cc:299] Negotiation successful
0928 14:57:04.468127 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 14:57:05.225196 6985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:05.224187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:32909 (local address 127.31.40.1:57229)
0928 14:57:05.224696 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:05.224898 (+ 202us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:32909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":243,"threads_started":1}
W20260928 14:57:05.225663 6625 heartbeater.cc:644] Failed to heartbeat to 127.31.40.62:32909 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.31.40.62:32909: connect: Connection refused (error 111)
I20260928 14:57:05.230741 6986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:05.229638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:32909 (local address 127.31.40.2:58889)
0928 14:57:05.230153 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:05.230323 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:32909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":275,"threads_started":1}
W20260928 14:57:05.231891 6774 heartbeater.cc:644] Failed to heartbeat to 127.31.40.62:32909 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.31.40.62:32909: connect: Connection refused (error 111)
I20260928 14:57:05.232254 6987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:05.230081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:32909 (local address 127.31.40.3:56789)
0928 14:57:05.231228 (+ 1147us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:05.231775 (+ 547us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:32909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1017,"thread_start_us":920,"threads_started":1}
W20260928 14:57:05.232661 6915 heartbeater.cc:644] Failed to heartbeat to 127.31.40.62:32909 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.31.40.62:32909: connect: Connection refused (error 111)
I20260928 14:57:05.745527 6988 raft_consensus.cc:482] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 40f51af6fc714868a4c4df423f897b6c)
I20260928 14:57:05.745884 6989 raft_consensus.cc:482] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 40f51af6fc714868a4c4df423f897b6c)
I20260928 14:57:05.745887 6988 raft_consensus.cc:504] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "40f51af6fc714868a4c4df423f897b6c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32909 } } peers { permanent_uuid: "791fc9b073e5437e8af2e1379eb4fe02" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 37667 } attrs { promote: false } } peers { permanent_uuid: "156238ac7d104bae8996c9dd47abd3cb" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 42157 } attrs { promote: false } }
I20260928 14:57:05.746165 6989 raft_consensus.cc:504] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "40f51af6fc714868a4c4df423f897b6c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32909 } } peers { permanent_uuid: "791fc9b073e5437e8af2e1379eb4fe02" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 37667 } attrs { promote: false } } peers { permanent_uuid: "156238ac7d104bae8996c9dd47abd3cb" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 42157 } attrs { promote: false } }
I20260928 14:57:05.747515 6988 leader_election.cc:288] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40f51af6fc714868a4c4df423f897b6c (127.31.40.62:32909), 156238ac7d104bae8996c9dd47abd3cb (127.31.40.60:42157)
I20260928 14:57:05.752794 6993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:05.748015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:32909 (local address 127.0.0.1:59854)
0928 14:57:05.752209 (+ 4194us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:05.752459 (+ 250us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:32909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4073,"threads_started":1}
W20260928 14:57:05.753696 6240 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.40.62:32909: connect: Connection refused (error 111)
I20260928 14:57:05.763238 6989 leader_election.cc:288] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40f51af6fc714868a4c4df423f897b6c (127.31.40.62:32909), 791fc9b073e5437e8af2e1379eb4fe02 (127.31.40.61:37667)
I20260928 14:57:05.765513 6990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:05.747637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:32909 (local address 127.0.0.1:59856)
0928 14:57:05.764179 (+ 16542us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:05.765195 (+ 1016us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:32909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16355,"thread_start_us":17220,"threads_started":1}
I20260928 14:57:05.765544 6993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:05.765020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:32909 (local address 127.0.0.1:59870)
0928 14:57:05.765248 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:05.765351 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:32909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
W20260928 14:57:05.765826 6240 leader_election.cc:334] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 40f51af6fc714868a4c4df423f897b6c (127.31.40.62:32909): Network error: Client connection negotiation failed: client connection to 127.31.40.62:32909: connect: Connection refused (error 111)
W20260928 14:57:05.766013 6429 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.40.62:32909: connect: Connection refused (error 111)
I20260928 14:57:05.768146 6990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:05.767675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:32909 (local address 127.0.0.1:59876)
0928 14:57:05.767825 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:05.767954 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:32909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 14:57:05.768468 6429 leader_election.cc:334] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 40f51af6fc714868a4c4df423f897b6c (127.31.40.62:32909): Network error: Client connection negotiation failed: client connection to 127.31.40.62:32909: connect: Connection refused (error 111)
I20260928 14:57:05.772218 6991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:05.747636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:42157 (local address 127.0.0.1:45198)
0928 14:57:05.751263 (+ 3627us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:05.751284 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:05.751430 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:05.765419 (+ 13989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:05.765424 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:05.765452 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:05.765838 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:05.765856 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:05.768125 (+ 2269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:05.768129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:05.769725 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:05.769738 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:05.771506 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:05.771511 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:05.771639 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:05.771665 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 14:57:05.771753 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 14:57:05.771857 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3512,"thread_start_us":3372,"threads_started":1}
I20260928 14:57:05.772372 6992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:05.747854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45198 (local address 127.31.40.60:42157)
0928 14:57:05.765062 (+ 17208us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:05.765068 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:05.765086 (+ 18us) server_negotiation.cc:407] Connection header received
0928 14:57:05.765162 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:05.765168 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:05.765246 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:05.766040 (+ 794us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:05.766357 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:05.767947 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:05.769992 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:05.771271 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:05.771405 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:05.771515 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:05.771806 (+ 291us) server_negotiation.cc:299] Negotiation successful
0928 14:57:05.771983 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17036,"thread_start_us":16894,"threads_started":1}
I20260928 14:57:05.772457 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:05.751609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.61:37667 (local address 127.0.0.1:58382)
0928 14:57:05.764842 (+ 13233us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:05.764860 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:05.765119 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:05.765843 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:05.765847 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:05.765875 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:05.766309 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:05.766328 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:05.768091 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:05.768096 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:05.769722 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:05.769736 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:05.771631 (+ 1895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:05.771635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:05.771821 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:05.771851 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 14:57:05.771956 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 14:57:05.772093 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13041,"mutex_wait_us":2,"thread_start_us":12811,"threads_started":1}
I20260928 14:57:05.773228 6995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:05.752231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58382 (local address 127.31.40.61:37667)
0928 14:57:05.764189 (+ 11958us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:05.764198 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:05.764980 (+ 782us) server_negotiation.cc:407] Connection header received
0928 14:57:05.765313 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:05.765320 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:05.765398 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:05.765473 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:05.766522 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:05.767929 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:05.769910 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:05.771263 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:05.771405 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:05.771517 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:05.772566 (+ 1049us) server_negotiation.cc:299] Negotiation successful
0928 14:57:05.772723 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11758,"thread_start_us":11572,"threads_started":1}
I20260928 14:57:05.773403 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "791fc9b073e5437e8af2e1379eb4fe02" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "156238ac7d104bae8996c9dd47abd3cb" is_pre_election: true
I20260928 14:57:05.773808 6456 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 791fc9b073e5437e8af2e1379eb4fe02 in term 1.
I20260928 14:57:05.773823 6266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "156238ac7d104bae8996c9dd47abd3cb" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "791fc9b073e5437e8af2e1379eb4fe02" is_pre_election: true
I20260928 14:57:05.774127 6266 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 156238ac7d104bae8996c9dd47abd3cb in term 1.
I20260928 14:57:05.774459 6242 leader_election.cc:302] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 156238ac7d104bae8996c9dd47abd3cb, 791fc9b073e5437e8af2e1379eb4fe02; no voters: 40f51af6fc714868a4c4df423f897b6c
I20260928 14:57:05.774935 6988 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 14:57:05.775125 6988 raft_consensus.cc:482] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 40f51af6fc714868a4c4df423f897b6c)
I20260928 14:57:05.775254 6988 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [term 1 FOLLOWER]: Advancing to term 2
I20260928 14:57:05.775254 6431 leader_election.cc:302] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 156238ac7d104bae8996c9dd47abd3cb, 791fc9b073e5437e8af2e1379eb4fe02; no voters: 40f51af6fc714868a4c4df423f897b6c
I20260928 14:57:05.775703 6989 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 14:57:05.775918 6989 raft_consensus.cc:482] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb [term 1 FOLLOWER]: Starting leader election (detected failure of leader 40f51af6fc714868a4c4df423f897b6c)
I20260928 14:57:05.776051 6989 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb [term 1 FOLLOWER]: Advancing to term 2
I20260928 14:57:05.778654 6988 raft_consensus.cc:504] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "40f51af6fc714868a4c4df423f897b6c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32909 } } peers { permanent_uuid: "791fc9b073e5437e8af2e1379eb4fe02" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 37667 } attrs { promote: false } } peers { permanent_uuid: "156238ac7d104bae8996c9dd47abd3cb" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 42157 } attrs { promote: false } }
I20260928 14:57:05.779238 6989 raft_consensus.cc:504] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "40f51af6fc714868a4c4df423f897b6c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32909 } } peers { permanent_uuid: "791fc9b073e5437e8af2e1379eb4fe02" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 37667 } attrs { promote: false } } peers { permanent_uuid: "156238ac7d104bae8996c9dd47abd3cb" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 42157 } attrs { promote: false } }
I20260928 14:57:05.780048 6989 leader_election.cc:288] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb [CANDIDATE]: Term 2 election: Requested vote from peers 40f51af6fc714868a4c4df423f897b6c (127.31.40.62:32909), 791fc9b073e5437e8af2e1379eb4fe02 (127.31.40.61:37667)
I20260928 14:57:05.780431 6991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:05.779833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:32909 (local address 127.0.0.1:59888)
0928 14:57:05.780069 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:05.780187 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:32909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260928 14:57:05.780558 6988 leader_election.cc:288] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [CANDIDATE]: Term 2 election: Requested vote from peers 40f51af6fc714868a4c4df423f897b6c (127.31.40.62:32909), 156238ac7d104bae8996c9dd47abd3cb (127.31.40.60:42157)
I20260928 14:57:05.780874 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:05.780393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:32909 (local address 127.0.0.1:59892)
0928 14:57:05.780574 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:05.780678 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:32909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 14:57:05.781128 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "791fc9b073e5437e8af2e1379eb4fe02" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "156238ac7d104bae8996c9dd47abd3cb"
I20260928 14:57:05.781175 6266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "156238ac7d104bae8996c9dd47abd3cb" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "791fc9b073e5437e8af2e1379eb4fe02"
I20260928 14:57:05.781443 6456 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 791fc9b073e5437e8af2e1379eb4fe02 in current term 2: Already voted for candidate 156238ac7d104bae8996c9dd47abd3cb in this term.
I20260928 14:57:05.781479 6266 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 156238ac7d104bae8996c9dd47abd3cb in current term 2: Already voted for candidate 791fc9b073e5437e8af2e1379eb4fe02 in this term.
I20260928 14:57:05.781596 6991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:05.781113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:32909 (local address 127.0.0.1:59906)
0928 14:57:05.781315 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:05.781429 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:32909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
W20260928 14:57:05.781741 6240 leader_election.cc:334] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 40f51af6fc714868a4c4df423f897b6c (127.31.40.62:32909): Network error: Client connection negotiation failed: client connection to 127.31.40.62:32909: connect: Connection refused (error 111)
I20260928 14:57:05.782169 6242 leader_election.cc:302] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 791fc9b073e5437e8af2e1379eb4fe02; no voters: 156238ac7d104bae8996c9dd47abd3cb, 40f51af6fc714868a4c4df423f897b6c
I20260928 14:57:05.782460 6988 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260928 14:57:05.782681 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:05.782274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:32909 (local address 127.0.0.1:59912)
0928 14:57:05.782428 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:05.782521 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:32909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 14:57:05.782828 6429 leader_election.cc:334] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 40f51af6fc714868a4c4df423f897b6c (127.31.40.62:32909): Network error: Client connection negotiation failed: client connection to 127.31.40.62:32909: connect: Connection refused (error 111)
I20260928 14:57:05.782958 6429 leader_election.cc:302] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 156238ac7d104bae8996c9dd47abd3cb; no voters: 40f51af6fc714868a4c4df423f897b6c, 791fc9b073e5437e8af2e1379eb4fe02
I20260928 14:57:05.783253 6989 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260928 14:57:06.227519 6998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:06.226575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:32909 (local address 127.31.40.1:33263)
0928 14:57:06.227142 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:06.227253 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:32909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":306,"threads_started":1}
I20260928 14:57:06.234357 6999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:06.232699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:32909 (local address 127.31.40.2:38313)
0928 14:57:06.234021 (+ 1322us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:06.234135 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:32909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1171,"thread_start_us":1062,"threads_started":1}
I20260928 14:57:06.235646 7000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:06.233360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:32909 (local address 127.31.40.3:54435)
0928 14:57:06.235308 (+ 1948us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:06.235409 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:32909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1827,"thread_start_us":1731,"threads_started":1}
I20260928 14:57:07.229614 7001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:07.228691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:32909 (local address 127.31.40.1:56129)
0928 14:57:07.229261 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:07.229385 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:32909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":332,"threads_started":1}
I20260928 14:57:07.236212 7002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:07.235291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:32909 (local address 127.31.40.2:60413)
0928 14:57:07.235846 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:07.236014 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:32909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":317,"threads_started":1}
I20260928 14:57:07.240510 7003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:07.236479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:32909 (local address 127.31.40.3:33061)
0928 14:57:07.240270 (+ 3791us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:07.240354 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:32909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3692,"thread_start_us":3600,"threads_started":1}
I20260928 14:57:07.799284 7004 raft_consensus.cc:482] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:57:07.799484 7004 raft_consensus.cc:504] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "40f51af6fc714868a4c4df423f897b6c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32909 } } peers { permanent_uuid: "791fc9b073e5437e8af2e1379eb4fe02" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 37667 } attrs { promote: false } } peers { permanent_uuid: "156238ac7d104bae8996c9dd47abd3cb" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 42157 } attrs { promote: false } }
I20260928 14:57:07.800181 7004 leader_election.cc:288] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 40f51af6fc714868a4c4df423f897b6c (127.31.40.62:32909), 156238ac7d104bae8996c9dd47abd3cb (127.31.40.60:42157)
I20260928 14:57:07.800765 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "791fc9b073e5437e8af2e1379eb4fe02" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "156238ac7d104bae8996c9dd47abd3cb" is_pre_election: true
I20260928 14:57:07.801048 6456 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 791fc9b073e5437e8af2e1379eb4fe02 in term 2.
I20260928 14:57:07.801347 7005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:07.800435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:32909 (local address 127.0.0.1:60204)
0928 14:57:07.800943 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:07.801126 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:32909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":275,"threads_started":1}
I20260928 14:57:07.801564 6242 leader_election.cc:302] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 156238ac7d104bae8996c9dd47abd3cb, 791fc9b073e5437e8af2e1379eb4fe02; no voters:
I20260928 14:57:07.801869 7004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260928 14:57:07.802017 7004 raft_consensus.cc:482] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:57:07.802132 7004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [term 2 FOLLOWER]: Advancing to term 3
I20260928 14:57:07.802999 7005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:07.802550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:32909 (local address 127.0.0.1:60214)
0928 14:57:07.802738 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:07.802833 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:32909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260928 14:57:07.803184 6240 leader_election.cc:334] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 40f51af6fc714868a4c4df423f897b6c (127.31.40.62:32909): Network error: Client connection negotiation failed: client connection to 127.31.40.62:32909: connect: Connection refused (error 111)
I20260928 14:57:07.804646 7004 raft_consensus.cc:504] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "40f51af6fc714868a4c4df423f897b6c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32909 } } peers { permanent_uuid: "791fc9b073e5437e8af2e1379eb4fe02" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 37667 } attrs { promote: false } } peers { permanent_uuid: "156238ac7d104bae8996c9dd47abd3cb" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 42157 } attrs { promote: false } }
I20260928 14:57:07.805269 7004 leader_election.cc:288] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [CANDIDATE]: Term 3 election: Requested vote from peers 40f51af6fc714868a4c4df423f897b6c (127.31.40.62:32909), 156238ac7d104bae8996c9dd47abd3cb (127.31.40.60:42157)
I20260928 14:57:07.805850 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "791fc9b073e5437e8af2e1379eb4fe02" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "156238ac7d104bae8996c9dd47abd3cb"
I20260928 14:57:07.806025 7005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:07.805544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:32909 (local address 127.0.0.1:60230)
0928 14:57:07.805748 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:07.805822 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:32909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 14:57:07.806059 6456 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb [term 2 FOLLOWER]: Advancing to term 3
I20260928 14:57:07.806972 7005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:07.806661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:32909 (local address 127.0.0.1:60242)
0928 14:57:07.806804 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:07.806880 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:32909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 14:57:07.807233 6240 leader_election.cc:334] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 40f51af6fc714868a4c4df423f897b6c (127.31.40.62:32909): Network error: Client connection negotiation failed: client connection to 127.31.40.62:32909: connect: Connection refused (error 111)
I20260928 14:57:07.808426 6456 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 791fc9b073e5437e8af2e1379eb4fe02 in term 3.
I20260928 14:57:07.808867 6242 leader_election.cc:302] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 156238ac7d104bae8996c9dd47abd3cb, 791fc9b073e5437e8af2e1379eb4fe02; no voters: 40f51af6fc714868a4c4df423f897b6c
I20260928 14:57:07.809242 7004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 14:57:07.809635 7004 raft_consensus.cc:686] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [term 3 LEADER]: Becoming Leader. State: Replica: 791fc9b073e5437e8af2e1379eb4fe02, State: Running, Role: LEADER
I20260928 14:57:07.810021 7004 consensus_queue.cc:237] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "40f51af6fc714868a4c4df423f897b6c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32909 } } peers { permanent_uuid: "791fc9b073e5437e8af2e1379eb4fe02" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 37667 } attrs { promote: false } } peers { permanent_uuid: "156238ac7d104bae8996c9dd47abd3cb" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 42157 } attrs { promote: false } }
I20260928 14:57:07.811816 7007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 791fc9b073e5437e8af2e1379eb4fe02. Latest consensus state: current_term: 3 leader_uuid: "791fc9b073e5437e8af2e1379eb4fe02" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "40f51af6fc714868a4c4df423f897b6c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32909 } } peers { permanent_uuid: "791fc9b073e5437e8af2e1379eb4fe02" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 37667 } attrs { promote: false } } peers { permanent_uuid: "156238ac7d104bae8996c9dd47abd3cb" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 42157 } attrs { promote: false } } }
I20260928 14:57:07.812238 7007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:07.812882 7009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:57:07.816437 7009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:57:07.817312 7009 catalog_manager.cc:1259] Loaded cluster ID: 913af42657514f2aa51a685dba5d7ba1
I20260928 14:57:07.817389 7009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:57:07.818602 7009 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:57:07.819389 7009 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02: Loaded TSK: 0
I20260928 14:57:07.820394 7009 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:57:07.852190 6256 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58378:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:57:07.860157 6256 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:57:07.864652 7005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:07.864242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:32909 (local address 127.0.0.1:60266)
0928 14:57:07.864418 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:07.864502 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:32909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 14:57:07.864904 6456 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb [term 3 FOLLOWER]: Refusing update from remote peer 791fc9b073e5437e8af2e1379eb4fe02: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260928 14:57:07.865760 7005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:07.865381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:32909 (local address 127.0.0.1:60280)
0928 14:57:07.865580 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:07.865651 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:32909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 14:57:07.865751 7007 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "156238ac7d104bae8996c9dd47abd3cb" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 42157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260928 14:57:07.866101 6240 consensus_peers.cc:487] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 -> Peer 40f51af6fc714868a4c4df423f897b6c (127.31.40.62:32909): Couldn't send request to peer 40f51af6fc714868a4c4df423f897b6c. Status: Network error: Client connection negotiation failed: client connection to 127.31.40.62:32909: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 14:57:07.869560 7011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 791fc9b073e5437e8af2e1379eb4fe02. Latest consensus state: current_term: 3 leader_uuid: "791fc9b073e5437e8af2e1379eb4fe02" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "40f51af6fc714868a4c4df423f897b6c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32909 } } peers { permanent_uuid: "791fc9b073e5437e8af2e1379eb4fe02" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 37667 } attrs { promote: false } } peers { permanent_uuid: "156238ac7d104bae8996c9dd47abd3cb" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 42157 } attrs { promote: false } } }
I20260928 14:57:07.869935 7011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:57:07.871307 7007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "791fc9b073e5437e8af2e1379eb4fe02" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "40f51af6fc714868a4c4df423f897b6c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32909 } } peers { permanent_uuid: "791fc9b073e5437e8af2e1379eb4fe02" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 37667 } attrs { promote: false } } peers { permanent_uuid: "156238ac7d104bae8996c9dd47abd3cb" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 42157 } attrs { promote: false } } }
I20260928 14:57:07.871644 7007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02 [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:07.872366 7011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "791fc9b073e5437e8af2e1379eb4fe02" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "40f51af6fc714868a4c4df423f897b6c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32909 } } peers { permanent_uuid: "791fc9b073e5437e8af2e1379eb4fe02" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 37667 } attrs { promote: false } } peers { permanent_uuid: "156238ac7d104bae8996c9dd47abd3cb" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 42157 } attrs { promote: false } } }
I20260928 14:57:07.872715 7011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 156238ac7d104bae8996c9dd47abd3cb [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:57:07.899942 7005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:07.880589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.2:33025 (local address 127.0.0.1:42954)
0928 14:57:07.880897 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:07.880931 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:07.881093 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:07.892788 (+ 11695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:07.892797 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:07.892841 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:07.893211 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:07.893227 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:07.894597 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:07.894606 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:07.897411 (+ 2805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:07.897435 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:07.898673 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:07.898683 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:07.898932 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:07.898968 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 14:57:07.899127 (+ 159us) client_negotiation.cc:240] Negotiation successful
0928 14:57:07.899363 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149}
I20260928 14:57:07.901110 7017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:07.880684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42954 (local address 127.31.40.2:33025)
0928 14:57:07.892463 (+ 11779us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:07.892467 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:07.892483 (+ 16us) server_negotiation.cc:407] Connection header received
0928 14:57:07.892540 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:07.892543 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:07.892606 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:07.892688 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:07.893397 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:07.894418 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:07.897635 (+ 3217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:07.898443 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:07.898633 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:07.898737 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:07.900563 (+ 1826us) server_negotiation.cc:299] Negotiation successful
0928 14:57:07.900754 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11683,"thread_start_us":11617,"threads_started":1}
I20260928 14:57:07.902042 6708 tablet_service.cc:1467] Processing CreateTablet for tablet 96de945906e842ab86f24a7244e787c8 (DEFAULT_TABLE table=test-table [id=3852fc2f67ec4442a3e1122dff848bcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:57:07.902554 6708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96de945906e842ab86f24a7244e787c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:07.907017 7019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:07.880845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.3:34255 (local address 127.0.0.1:32808)
0928 14:57:07.889741 (+ 8896us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:07.889773 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:07.889962 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:07.900734 (+ 10772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:07.900739 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:07.900770 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:07.901151 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:07.901166 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:07.902482 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:07.902489 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:07.904711 (+ 2222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:07.904724 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:07.906051 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:07.906059 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:07.906217 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:07.906247 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 14:57:07.906343 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 14:57:07.906561 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8673,"mutex_wait_us":27,"thread_start_us":10894,"threads_started":1}
I20260928 14:57:07.907156 7018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:07.881323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:35061 (local address 127.0.0.1:46394)
0928 14:57:07.892008 (+ 10685us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:07.892024 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:07.892127 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:07.899540 (+ 7413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:07.899546 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:07.899570 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:07.900076 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:07.900104 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:07.901540 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:07.901544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:07.903649 (+ 2105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:07.903668 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:07.905862 (+ 2194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:07.905872 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:07.906101 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:07.906142 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 14:57:07.906255 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 14:57:07.906458 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10560,"mutex_wait_us":12,"thread_start_us":8770,"threads_started":1}
I20260928 14:57:07.907987 7020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:07.881410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32808 (local address 127.31.40.3:34255)
0928 14:57:07.900374 (+ 18964us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:07.900380 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:07.900407 (+ 27us) server_negotiation.cc:407] Connection header received
0928 14:57:07.900474 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:07.900482 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:07.900573 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:07.900695 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:07.901396 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:07.902330 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:07.904876 (+ 2546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:07.905817 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:07.905963 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:07.906085 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:07.907363 (+ 1278us) server_negotiation.cc:299] Negotiation successful
0928 14:57:07.907526 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18836,"thread_start_us":18708,"threads_started":1}
I20260928 14:57:07.908821 6849 tablet_service.cc:1467] Processing CreateTablet for tablet 96de945906e842ab86f24a7244e787c8 (DEFAULT_TABLE table=test-table [id=3852fc2f67ec4442a3e1122dff848bcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:57:07.908946 7021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:07.881810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46394 (local address 127.31.40.1:35061)
0928 14:57:07.899205 (+ 17395us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:07.899211 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:07.899233 (+ 22us) server_negotiation.cc:407] Connection header received
0928 14:57:07.899303 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:07.899308 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:07.899383 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:07.900272 (+ 889us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:07.900446 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:07.901367 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:07.904314 (+ 2947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:07.905611 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:07.905768 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:07.905865 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:07.908200 (+ 2335us) server_negotiation.cc:299] Negotiation successful
0928 14:57:07.908349 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17202,"thread_start_us":17066,"threads_started":1}
I20260928 14:57:07.909318 6849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96de945906e842ab86f24a7244e787c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:07.909876 7022 tablet_bootstrap.cc:492] T 96de945906e842ab86f24a7244e787c8 P 1484aa0f119348a1af94cea1a988d0f2: Bootstrap starting.
I20260928 14:57:07.911531 7022 tablet_bootstrap.cc:654] T 96de945906e842ab86f24a7244e787c8 P 1484aa0f119348a1af94cea1a988d0f2: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:07.912168 6559 tablet_service.cc:1467] Processing CreateTablet for tablet 96de945906e842ab86f24a7244e787c8 (DEFAULT_TABLE table=test-table [id=3852fc2f67ec4442a3e1122dff848bcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:57:07.912631 6559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96de945906e842ab86f24a7244e787c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:07.917362 7024 tablet_bootstrap.cc:492] T 96de945906e842ab86f24a7244e787c8 P f2f0f28f0299472386abafe15704df7b: Bootstrap starting.
I20260928 14:57:07.917726 7022 tablet_bootstrap.cc:492] T 96de945906e842ab86f24a7244e787c8 P 1484aa0f119348a1af94cea1a988d0f2: No bootstrap required, opened a new log
I20260928 14:57:07.917901 7022 ts_tablet_manager.cc:1397] T 96de945906e842ab86f24a7244e787c8 P 1484aa0f119348a1af94cea1a988d0f2: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.008s
I20260928 14:57:07.918877 7022 raft_consensus.cc:348] T 96de945906e842ab86f24a7244e787c8 P 1484aa0f119348a1af94cea1a988d0f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1484aa0f119348a1af94cea1a988d0f2" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 33025 } } peers { permanent_uuid: "f2f0f28f0299472386abafe15704df7b" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34255 } } peers { permanent_uuid: "50bbf97992ac4d92a0ce12b7b1be3bbe" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 35061 } }
I20260928 14:57:07.919164 7022 raft_consensus.cc:374] T 96de945906e842ab86f24a7244e787c8 P 1484aa0f119348a1af94cea1a988d0f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:07.919278 7022 raft_consensus.cc:729] T 96de945906e842ab86f24a7244e787c8 P 1484aa0f119348a1af94cea1a988d0f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1484aa0f119348a1af94cea1a988d0f2, State: Initialized, Role: FOLLOWER
I20260928 14:57:07.919587 7022 consensus_queue.cc:260] T 96de945906e842ab86f24a7244e787c8 P 1484aa0f119348a1af94cea1a988d0f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1484aa0f119348a1af94cea1a988d0f2" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 33025 } } peers { permanent_uuid: "f2f0f28f0299472386abafe15704df7b" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34255 } } peers { permanent_uuid: "50bbf97992ac4d92a0ce12b7b1be3bbe" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 35061 } }
I20260928 14:57:07.919838 7024 tablet_bootstrap.cc:654] T 96de945906e842ab86f24a7244e787c8 P f2f0f28f0299472386abafe15704df7b: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:07.920588 7025 tablet_bootstrap.cc:492] T 96de945906e842ab86f24a7244e787c8 P 50bbf97992ac4d92a0ce12b7b1be3bbe: Bootstrap starting.
I20260928 14:57:07.920722 7022 ts_tablet_manager.cc:1428] T 96de945906e842ab86f24a7244e787c8 P 1484aa0f119348a1af94cea1a988d0f2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 14:57:07.922675 7025 tablet_bootstrap.cc:654] T 96de945906e842ab86f24a7244e787c8 P 50bbf97992ac4d92a0ce12b7b1be3bbe: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:07.927286 7028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:07.921471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:32909 (local address 127.31.40.2:51599)
0928 14:57:07.926549 (+ 5078us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:07.926940 (+ 391us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:32909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4916,"thread_start_us":4768,"threads_started":1}
I20260928 14:57:07.929374 6775 heartbeater.cc:503] Master 127.31.40.61:37667 was elected leader, sending a full tablet report...
I20260928 14:57:07.929945 7028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:07.928032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:32909 (local address 127.31.40.2:32791)
0928 14:57:07.928621 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:07.928704 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:32909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":487,"threads_started":1}
I20260928 14:57:07.932253 7025 tablet_bootstrap.cc:492] T 96de945906e842ab86f24a7244e787c8 P 50bbf97992ac4d92a0ce12b7b1be3bbe: No bootstrap required, opened a new log
I20260928 14:57:07.932437 7025 ts_tablet_manager.cc:1397] T 96de945906e842ab86f24a7244e787c8 P 50bbf97992ac4d92a0ce12b7b1be3bbe: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.007s
I20260928 14:57:07.932632 7024 tablet_bootstrap.cc:492] T 96de945906e842ab86f24a7244e787c8 P f2f0f28f0299472386abafe15704df7b: No bootstrap required, opened a new log
I20260928 14:57:07.932806 7024 ts_tablet_manager.cc:1397] T 96de945906e842ab86f24a7244e787c8 P f2f0f28f0299472386abafe15704df7b: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.007s
I20260928 14:57:07.933421 7025 raft_consensus.cc:348] T 96de945906e842ab86f24a7244e787c8 P 50bbf97992ac4d92a0ce12b7b1be3bbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1484aa0f119348a1af94cea1a988d0f2" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 33025 } } peers { permanent_uuid: "f2f0f28f0299472386abafe15704df7b" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34255 } } peers { permanent_uuid: "50bbf97992ac4d92a0ce12b7b1be3bbe" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 35061 } }
I20260928 14:57:07.933688 7025 raft_consensus.cc:374] T 96de945906e842ab86f24a7244e787c8 P 50bbf97992ac4d92a0ce12b7b1be3bbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:07.933799 7025 raft_consensus.cc:729] T 96de945906e842ab86f24a7244e787c8 P 50bbf97992ac4d92a0ce12b7b1be3bbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50bbf97992ac4d92a0ce12b7b1be3bbe, State: Initialized, Role: FOLLOWER
I20260928 14:57:07.933684 7024 raft_consensus.cc:348] T 96de945906e842ab86f24a7244e787c8 P f2f0f28f0299472386abafe15704df7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1484aa0f119348a1af94cea1a988d0f2" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 33025 } } peers { permanent_uuid: "f2f0f28f0299472386abafe15704df7b" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34255 } } peers { permanent_uuid: "50bbf97992ac4d92a0ce12b7b1be3bbe" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 35061 } }
I20260928 14:57:07.933960 7024 raft_consensus.cc:374] T 96de945906e842ab86f24a7244e787c8 P f2f0f28f0299472386abafe15704df7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:07.934054 7024 raft_consensus.cc:729] T 96de945906e842ab86f24a7244e787c8 P f2f0f28f0299472386abafe15704df7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2f0f28f0299472386abafe15704df7b, State: Initialized, Role: FOLLOWER
I20260928 14:57:07.934074 7025 consensus_queue.cc:260] T 96de945906e842ab86f24a7244e787c8 P 50bbf97992ac4d92a0ce12b7b1be3bbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1484aa0f119348a1af94cea1a988d0f2" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 33025 } } peers { permanent_uuid: "f2f0f28f0299472386abafe15704df7b" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34255 } } peers { permanent_uuid: "50bbf97992ac4d92a0ce12b7b1be3bbe" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 35061 } }
I20260928 14:57:07.934312 7024 consensus_queue.cc:260] T 96de945906e842ab86f24a7244e787c8 P f2f0f28f0299472386abafe15704df7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1484aa0f119348a1af94cea1a988d0f2" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 33025 } } peers { permanent_uuid: "f2f0f28f0299472386abafe15704df7b" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34255 } } peers { permanent_uuid: "50bbf97992ac4d92a0ce12b7b1be3bbe" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 35061 } }
I20260928 14:57:07.935034 7025 ts_tablet_manager.cc:1428] T 96de945906e842ab86f24a7244e787c8 P 50bbf97992ac4d92a0ce12b7b1be3bbe: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 14:57:07.937484 6626 heartbeater.cc:503] Master 127.31.40.61:37667 was elected leader, sending a full tablet report...
I20260928 14:57:07.939527 7033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:07.935302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:32909 (local address 127.31.40.1:55711)
0928 14:57:07.935974 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:07.938165 (+ 2191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:32909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":345,"threads_started":1}
I20260928 14:57:07.941025 7033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:07.940470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:32909 (local address 127.31.40.1:55933)
0928 14:57:07.940824 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:07.940913 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:32909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243}
I20260928 14:57:07.942406 7024 ts_tablet_manager.cc:1428] T 96de945906e842ab86f24a7244e787c8 P f2f0f28f0299472386abafe15704df7b: Time spent starting tablet: real 0.009s user 0.002s sys 0.007s
I20260928 14:57:07.944643 6916 heartbeater.cc:503] Master 127.31.40.61:37667 was elected leader, sending a full tablet report...
I20260928 14:57:07.944759 7034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:07.942533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:32909 (local address 127.31.40.3:52589)
0928 14:57:07.944258 (+ 1725us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:07.944423 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:32909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1555,"thread_start_us":1406,"threads_started":1}
I20260928 14:57:07.946166 7034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:07.945679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:32909 (local address 127.31.40.3:42381)
0928 14:57:07.945880 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:07.945981 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:32909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 14:57:07.987497 7031 raft_consensus.cc:482] T 96de945906e842ab86f24a7244e787c8 P 50bbf97992ac4d92a0ce12b7b1be3bbe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:57:07.987805 7031 raft_consensus.cc:504] T 96de945906e842ab86f24a7244e787c8 P 50bbf97992ac4d92a0ce12b7b1be3bbe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1484aa0f119348a1af94cea1a988d0f2" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 33025 } } peers { permanent_uuid: "f2f0f28f0299472386abafe15704df7b" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34255 } } peers { permanent_uuid: "50bbf97992ac4d92a0ce12b7b1be3bbe" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 35061 } }
I20260928 14:57:07.988590 7031 leader_election.cc:288] T 96de945906e842ab86f24a7244e787c8 P 50bbf97992ac4d92a0ce12b7b1be3bbe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1484aa0f119348a1af94cea1a988d0f2 (127.31.40.2:33025), f2f0f28f0299472386abafe15704df7b (127.31.40.3:34255)
I20260928 14:57:07.989233 6728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96de945906e842ab86f24a7244e787c8" candidate_uuid: "50bbf97992ac4d92a0ce12b7b1be3bbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1484aa0f119348a1af94cea1a988d0f2" is_pre_election: true
I20260928 14:57:07.989269 6869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96de945906e842ab86f24a7244e787c8" candidate_uuid: "50bbf97992ac4d92a0ce12b7b1be3bbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2f0f28f0299472386abafe15704df7b" is_pre_election: true
I20260928 14:57:07.989513 6728 raft_consensus.cc:2454] T 96de945906e842ab86f24a7244e787c8 P 1484aa0f119348a1af94cea1a988d0f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50bbf97992ac4d92a0ce12b7b1be3bbe in term 0.
I20260928 14:57:07.989533 6869 raft_consensus.cc:2454] T 96de945906e842ab86f24a7244e787c8 P f2f0f28f0299472386abafe15704df7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50bbf97992ac4d92a0ce12b7b1be3bbe in term 0.
I20260928 14:57:07.990043 6515 leader_election.cc:302] T 96de945906e842ab86f24a7244e787c8 P 50bbf97992ac4d92a0ce12b7b1be3bbe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1484aa0f119348a1af94cea1a988d0f2, 50bbf97992ac4d92a0ce12b7b1be3bbe; no voters:
I20260928 14:57:07.990358 7031 raft_consensus.cc:2781] T 96de945906e842ab86f24a7244e787c8 P 50bbf97992ac4d92a0ce12b7b1be3bbe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:57:07.990479 7031 raft_consensus.cc:482] T 96de945906e842ab86f24a7244e787c8 P 50bbf97992ac4d92a0ce12b7b1be3bbe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:57:07.990578 7031 raft_consensus.cc:3037] T 96de945906e842ab86f24a7244e787c8 P 50bbf97992ac4d92a0ce12b7b1be3bbe [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:07.992777 7031 raft_consensus.cc:504] T 96de945906e842ab86f24a7244e787c8 P 50bbf97992ac4d92a0ce12b7b1be3bbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1484aa0f119348a1af94cea1a988d0f2" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 33025 } } peers { permanent_uuid: "f2f0f28f0299472386abafe15704df7b" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34255 } } peers { permanent_uuid: "50bbf97992ac4d92a0ce12b7b1be3bbe" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 35061 } }
I20260928 14:57:07.993391 7031 leader_election.cc:288] T 96de945906e842ab86f24a7244e787c8 P 50bbf97992ac4d92a0ce12b7b1be3bbe [CANDIDATE]: Term 1 election: Requested vote from peers 1484aa0f119348a1af94cea1a988d0f2 (127.31.40.2:33025), f2f0f28f0299472386abafe15704df7b (127.31.40.3:34255)
I20260928 14:57:07.993984 6728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96de945906e842ab86f24a7244e787c8" candidate_uuid: "50bbf97992ac4d92a0ce12b7b1be3bbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1484aa0f119348a1af94cea1a988d0f2"
I20260928 14:57:07.994024 6869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96de945906e842ab86f24a7244e787c8" candidate_uuid: "50bbf97992ac4d92a0ce12b7b1be3bbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2f0f28f0299472386abafe15704df7b"
I20260928 14:57:07.994218 6728 raft_consensus.cc:3037] T 96de945906e842ab86f24a7244e787c8 P 1484aa0f119348a1af94cea1a988d0f2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:07.994233 6869 raft_consensus.cc:3037] T 96de945906e842ab86f24a7244e787c8 P f2f0f28f0299472386abafe15704df7b [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:07.996347 6728 raft_consensus.cc:2454] T 96de945906e842ab86f24a7244e787c8 P 1484aa0f119348a1af94cea1a988d0f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50bbf97992ac4d92a0ce12b7b1be3bbe in term 1.
I20260928 14:57:07.996351 6869 raft_consensus.cc:2454] T 96de945906e842ab86f24a7244e787c8 P f2f0f28f0299472386abafe15704df7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50bbf97992ac4d92a0ce12b7b1be3bbe in term 1.
I20260928 14:57:07.996861 6515 leader_election.cc:302] T 96de945906e842ab86f24a7244e787c8 P 50bbf97992ac4d92a0ce12b7b1be3bbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1484aa0f119348a1af94cea1a988d0f2, 50bbf97992ac4d92a0ce12b7b1be3bbe, f2f0f28f0299472386abafe15704df7b; no voters:
I20260928 14:57:07.997184 7031 raft_consensus.cc:2781] T 96de945906e842ab86f24a7244e787c8 P 50bbf97992ac4d92a0ce12b7b1be3bbe [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:57:07.997612 7031 raft_consensus.cc:686] T 96de945906e842ab86f24a7244e787c8 P 50bbf97992ac4d92a0ce12b7b1be3bbe [term 1 LEADER]: Becoming Leader. State: Replica: 50bbf97992ac4d92a0ce12b7b1be3bbe, State: Running, Role: LEADER
I20260928 14:57:07.998272 7031 consensus_queue.cc:237] T 96de945906e842ab86f24a7244e787c8 P 50bbf97992ac4d92a0ce12b7b1be3bbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1484aa0f119348a1af94cea1a988d0f2" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 33025 } } peers { permanent_uuid: "f2f0f28f0299472386abafe15704df7b" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34255 } } peers { permanent_uuid: "50bbf97992ac4d92a0ce12b7b1be3bbe" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 35061 } }
I20260928 14:57:07.998780 7033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:07.998276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:32909 (local address 127.31.40.1:47785)
0928 14:57:07.998467 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:07.998592 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:32909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 14:57:08.001927 6255 catalog_manager.cc:5495] T 96de945906e842ab86f24a7244e787c8 P 50bbf97992ac4d92a0ce12b7b1be3bbe reported cstate change: term changed from 0 to 1, leader changed from <none> to 50bbf97992ac4d92a0ce12b7b1be3bbe (127.31.40.1). New cstate: current_term: 1 leader_uuid: "50bbf97992ac4d92a0ce12b7b1be3bbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1484aa0f119348a1af94cea1a988d0f2" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 33025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2f0f28f0299472386abafe15704df7b" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50bbf97992ac4d92a0ce12b7b1be3bbe" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 35061 } health_report { overall_health: HEALTHY } } }
W20260928 14:57:08.010483 6627 tablet.cc:2369] T 96de945906e842ab86f24a7244e787c8 P 50bbf97992ac4d92a0ce12b7b1be3bbe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:57:08.036754 7021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:08.031702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46406 (local address 127.31.40.1:35061)
0928 14:57:08.031969 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:08.031973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:08.031993 (+ 20us) server_negotiation.cc:407] Connection header received
0928 14:57:08.032222 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:08.032227 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:08.032301 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:08.032440 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:57:08.032762 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:08.033611 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:08.035143 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:08.035552 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:08.035684 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:08.036102 (+ 418us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:57:08.036209 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:57:08.036279 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:08.036475 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 14:57:08.036545 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 14:57:08.040062 6869 raft_consensus.cc:1261] T 96de945906e842ab86f24a7244e787c8 P f2f0f28f0299472386abafe15704df7b [term 1 FOLLOWER]: Refusing update from remote peer 50bbf97992ac4d92a0ce12b7b1be3bbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:57:08.040199 6728 raft_consensus.cc:1261] T 96de945906e842ab86f24a7244e787c8 P 1484aa0f119348a1af94cea1a988d0f2 [term 1 FOLLOWER]: Refusing update from remote peer 50bbf97992ac4d92a0ce12b7b1be3bbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:57:08.040876 7031 consensus_queue.cc:1035] T 96de945906e842ab86f24a7244e787c8 P 50bbf97992ac4d92a0ce12b7b1be3bbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2f0f28f0299472386abafe15704df7b" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:57:08.041306 7035 consensus_queue.cc:1035] T 96de945906e842ab86f24a7244e787c8 P 50bbf97992ac4d92a0ce12b7b1be3bbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "1484aa0f119348a1af94cea1a988d0f2" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 33025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:57:08.043938 7028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:08.043422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:32909 (local address 127.31.40.2:58931)
0928 14:57:08.043572 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:08.043714 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:32909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 14:57:08.044312 7034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:08.043776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:32909 (local address 127.31.40.3:54249)
0928 14:57:08.044044 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:08.044152 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:32909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
I20260928 14:57:08.049983 7028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:08.049435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:32909 (local address 127.31.40.2:40535)
0928 14:57:08.049709 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:08.049808 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:32909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174}
I20260928 14:57:08.056085 7033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:08.055564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:32909 (local address 127.31.40.1:42689)
0928 14:57:08.055726 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:08.055846 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:32909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 14:57:08.064033 6254 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:57:08.064551 6254 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:57:08.065228 7033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:08.063556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:32909 (local address 127.31.40.1:36875)
0928 14:57:08.064933 (+ 1377us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:08.065030 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:32909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1252}
I20260928 14:57:08.065541 7034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:08.064992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:32909 (local address 127.31.40.3:36021)
0928 14:57:08.065274 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:08.065376 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:32909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172}
I20260928 14:57:08.066712 7039 mvcc.cc:205] Tried to move back new op lower bound from 7334328025245388800 to 7334328025083572224. Current Snapshot: MvccSnapshot[applied={T|T < 7334328025245388800}]
I20260928 14:57:08.072362 6254 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 791fc9b073e5437e8af2e1379eb4fe02: Sending DeleteTablet for 3 replicas of tablet 96de945906e842ab86f24a7244e787c8
I20260928 14:57:08.073544 6849 tablet_service.cc:1514] Processing DeleteTablet for tablet 96de945906e842ab86f24a7244e787c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:57:08 UTC) from {username='oryx', principal='oryx/127.31.40.61@KRBTEST.COM'} at 127.0.0.1:32808
I20260928 14:57:08.073649 6708 tablet_service.cc:1514] Processing DeleteTablet for tablet 96de945906e842ab86f24a7244e787c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:57:08 UTC) from {username='oryx', principal='oryx/127.31.40.61@KRBTEST.COM'} at 127.0.0.1:42954
I20260928 14:57:08.074308 7046 tablet_replica.cc:330] T 96de945906e842ab86f24a7244e787c8 P f2f0f28f0299472386abafe15704df7b: stopping tablet replica
I20260928 14:57:08.074467 7047 tablet_replica.cc:330] T 96de945906e842ab86f24a7244e787c8 P 1484aa0f119348a1af94cea1a988d0f2: stopping tablet replica
I20260928 14:57:08.074638 7046 raft_consensus.cc:2229] T 96de945906e842ab86f24a7244e787c8 P f2f0f28f0299472386abafe15704df7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:57:08.074703 7047 raft_consensus.cc:2229] T 96de945906e842ab86f24a7244e787c8 P 1484aa0f119348a1af94cea1a988d0f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:57:08.074949 6559 tablet_service.cc:1514] Processing DeleteTablet for tablet 96de945906e842ab86f24a7244e787c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:57:08 UTC) from {username='oryx', principal='oryx/127.31.40.61@KRBTEST.COM'} at 127.0.0.1:46394
I20260928 14:57:08.075050 7046 raft_consensus.cc:2258] T 96de945906e842ab86f24a7244e787c8 P f2f0f28f0299472386abafe15704df7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:57:08.075616 7047 raft_consensus.cc:2258] T 96de945906e842ab86f24a7244e787c8 P 1484aa0f119348a1af94cea1a988d0f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:57:08.075999 7048 tablet_replica.cc:330] T 96de945906e842ab86f24a7244e787c8 P 50bbf97992ac4d92a0ce12b7b1be3bbe: stopping tablet replica
I20260928 14:57:08.076236 7048 raft_consensus.cc:2229] T 96de945906e842ab86f24a7244e787c8 P 50bbf97992ac4d92a0ce12b7b1be3bbe [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:57:08.076548 7046 ts_tablet_manager.cc:1905] T 96de945906e842ab86f24a7244e787c8 P f2f0f28f0299472386abafe15704df7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:57:08.076843 7048 raft_consensus.cc:2258] T 96de945906e842ab86f24a7244e787c8 P 50bbf97992ac4d92a0ce12b7b1be3bbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:57:08.078001 7048 ts_tablet_manager.cc:1905] T 96de945906e842ab86f24a7244e787c8 P 50bbf97992ac4d92a0ce12b7b1be3bbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:57:08.078181 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 6446
I20260928 14:57:08.078805 7047 ts_tablet_manager.cc:1905] T 96de945906e842ab86f24a7244e787c8 P 1484aa0f119348a1af94cea1a988d0f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:57:08.085278 7048 ts_tablet_manager.cc:1918] T 96de945906e842ab86f24a7244e787c8 P 50bbf97992ac4d92a0ce12b7b1be3bbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:57:08.085505 7048 log.cc:1196] T 96de945906e842ab86f24a7244e787c8 P 50bbf97992ac4d92a0ce12b7b1be3bbe: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-0/wal/wals/96de945906e842ab86f24a7244e787c8
I20260928 14:57:08.085834 7047 ts_tablet_manager.cc:1918] T 96de945906e842ab86f24a7244e787c8 P 1484aa0f119348a1af94cea1a988d0f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:57:08.085894 7046 ts_tablet_manager.cc:1918] T 96de945906e842ab86f24a7244e787c8 P f2f0f28f0299472386abafe15704df7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:57:08.085963 7048 ts_tablet_manager.cc:1939] T 96de945906e842ab86f24a7244e787c8 P 50bbf97992ac4d92a0ce12b7b1be3bbe: Deleting consensus metadata
I20260928 14:57:08.086086 7046 log.cc:1196] T 96de945906e842ab86f24a7244e787c8 P f2f0f28f0299472386abafe15704df7b: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-2/wal/wals/96de945906e842ab86f24a7244e787c8
I20260928 14:57:08.086138 7047 log.cc:1196] T 96de945906e842ab86f24a7244e787c8 P 1484aa0f119348a1af94cea1a988d0f2: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607365362981-31904-0/minicluster-data/ts-1/wal/wals/96de945906e842ab86f24a7244e787c8
I20260928 14:57:08.086688 7047 ts_tablet_manager.cc:1939] T 96de945906e842ab86f24a7244e787c8 P 1484aa0f119348a1af94cea1a988d0f2: Deleting consensus metadata
I20260928 14:57:08.086709 7046 ts_tablet_manager.cc:1939] T 96de945906e842ab86f24a7244e787c8 P f2f0f28f0299472386abafe15704df7b: Deleting consensus metadata
I20260928 14:57:08.087288 6240 catalog_manager.cc:4841] TS 50bbf97992ac4d92a0ce12b7b1be3bbe (127.31.40.1:35061): tablet 96de945906e842ab86f24a7244e787c8 (table test-table [id=3852fc2f67ec4442a3e1122dff848bcc]) successfully deleted
I20260928 14:57:08.087944 6243 catalog_manager.cc:4841] TS 1484aa0f119348a1af94cea1a988d0f2 (127.31.40.2:33025): tablet 96de945906e842ab86f24a7244e787c8 (table test-table [id=3852fc2f67ec4442a3e1122dff848bcc]) successfully deleted
I20260928 14:57:08.088204 6242 catalog_manager.cc:4841] TS f2f0f28f0299472386abafe15704df7b (127.31.40.3:34255): tablet 96de945906e842ab86f24a7244e787c8 (table test-table [id=3852fc2f67ec4442a3e1122dff848bcc]) successfully deleted
I20260928 14:57:08.102293 7021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:08.079641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46408 (local address 127.31.40.1:35061)
0928 14:57:08.079833 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:08.079840 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:08.088410 (+ 8570us) server_negotiation.cc:407] Connection header received
0928 14:57:08.088635 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:08.088641 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:08.088718 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:08.088825 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:08.090167 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:08.090903 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:08.092242 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:08.092660 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:08.092803 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:08.095391 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:08.095407 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:08.095425 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:08.095465 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:08.098847 (+ 3382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:08.099743 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:08.099758 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:08.099761 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:08.100038 (+ 277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:08.100503 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:08.100509 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:08.100513 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:08.100936 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:08.101289 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:08.101629 (+ 340us) server_negotiation.cc:299] Negotiation successful
0928 14:57:08.101899 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 14:57:08.103281 6619 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:57:08.239455 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 6446
I20260928 14:57:08.263824 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 6636
I20260928 14:57:08.278929 7017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:08.264742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42966 (local address 127.31.40.2:33025)
0928 14:57:08.264931 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:08.264936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:08.264955 (+ 19us) server_negotiation.cc:407] Connection header received
0928 14:57:08.265075 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:08.265080 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:08.265150 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:08.265245 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:08.266747 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:08.267784 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:08.269284 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:08.269707 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:08.269852 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:08.272646 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:08.272662 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:08.272696 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:08.272731 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:08.275802 (+ 3071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:08.276744 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:08.276752 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:08.276754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:08.276885 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:08.277334 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:08.277338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:08.277340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:08.277680 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:08.278003 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:08.278225 (+ 222us) server_negotiation.cc:299] Negotiation successful
0928 14:57:08.278421 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 14:57:08.279667 6768 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:57:08.297772 7018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:08.297279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:32909 (local address 127.0.0.1:36604)
0928 14:57:08.297518 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:08.297626 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:32909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 14:57:08.298806 7018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:08.298448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:32909 (local address 127.0.0.1:36606)
0928 14:57:08.298598 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:08.298674 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:32909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 14:57:08.376567 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 6636
I20260928 14:57:08.396203 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 6786
I20260928 14:57:08.412153 7020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:08.397080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37188 (local address 127.31.40.3:34255)
0928 14:57:08.397305 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:08.397313 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:08.397337 (+ 24us) server_negotiation.cc:407] Connection header received
0928 14:57:08.397590 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:08.397599 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:08.397677 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:08.397800 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:08.399311 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:08.400341 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:08.401738 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:08.402223 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:08.402375 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:08.405385 (+ 3010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:08.405403 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:08.405422 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:08.405468 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:08.408875 (+ 3407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:08.409681 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:08.409690 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:08.409693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:08.409871 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:08.410321 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:08.410327 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:08.410331 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:08.410757 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:08.411129 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:08.411483 (+ 354us) server_negotiation.cc:299] Negotiation successful
0928 14:57:08.411771 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 14:57:08.412878 6909 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:57:08.509925 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 6786
I20260928 14:57:08.528235 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 6117
I20260928 14:57:08.529422 6286 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:57:08.656486 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 6117
W20260928 14:57:08.679432 6430 connection.cc:535] server connection from 127.0.0.1:45198 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 14:57:08.679795 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 6219
I20260928 14:57:08.681254 6476 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:57:08.798434 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 6219
2026-09-28T14:57:08Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (11947 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:57:08 dist-test-slave-j968 krb5kdc[7057](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:57:08 dist-test-slave-j968 krb5kdc[7057](info): set up 2 sockets
Sep 28 14:57:08 dist-test-slave-j968 krb5kdc[7057](info): commencing operation
krb5kdc: starting...
W20260928 14:57:09.914085 31904 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Sep 28 14:57:10 dist-test-slave-j968 krb5kdc[7057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:57:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:57:10Z Disabled control of system clock
WARNING: no policy specified for oryx/127.31.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/krb5kdc/oryx_127.31.40.62.keytab.
Entry for principal oryx/127.31.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/krb5kdc/oryx_127.31.40.62.keytab.
I20260928 14:57:10.160079 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:32977
--webserver_interface=127.31.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/krb5kdc/oryx_127.31.40.62.keytab
--principal=oryx/127.31.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:34277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.40.62:32977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:57:10.498453 7070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:10.498768 7070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:10.498867 7070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:10.506327 7070 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:57:10.506453 7070 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 14:57:10.506496 7070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:10.506554 7070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:10.506604 7070 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:57:10.506665 7070 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:57:10.516762 7070 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:34277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.40.62:32977
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/krb5kdc/oryx_127.31.40.62.keytab
--principal=oryx/127.31.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:32977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:57:10.518050 7070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:10.519313 7070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:10.532245 7075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:10.532229 7078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:10.532428 7070 server_base.cc:1034] running on GCE node
W20260928 14:57:10.532181 7076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:10.533667 7070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:10.535109 7070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:10.536334 7070 hybrid_clock.cc:648] HybridClock initialized: now 1790607430536275 us; error 49 us; skew 500 ppm
I20260928 14:57:10.536717 7070 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:57:10 dist-test-slave-j968 krb5kdc[7057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607430, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:10.540053 7070 init.cc:377] Logged in from keytab as oryx/127.31.40.62@KRBTEST.COM (short username oryx)
I20260928 14:57:10.541765 7070 webserver.cc:460] Webserver started at http://127.31.40.62:43751/ using document root <none> and password file <none>
I20260928 14:57:10.542618 7070 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:10.542714 7070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:10.543066 7070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:10.546739 7070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/master-0/data/instance:
uuid: "73109dfa9b4d4463a096a15f0501a28c"
format_stamp: "Formatted at 2026-09-28 14:57:10 on dist-test-slave-j968"
server_key: "dc5b76cff65eb2db69292276f53d7a23"
server_key_iv: "0aca73e9ab7105923276edac30d6057a"
server_key_version: "encryptionkey@0"
I20260928 14:57:10.547681 7070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/master-0/wal/instance:
uuid: "73109dfa9b4d4463a096a15f0501a28c"
format_stamp: "Formatted at 2026-09-28 14:57:10 on dist-test-slave-j968"
server_key: "dc5b76cff65eb2db69292276f53d7a23"
server_key_iv: "0aca73e9ab7105923276edac30d6057a"
server_key_version: "encryptionkey@0"
I20260928 14:57:10.553473 7070 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 14:57:10.557348 7085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:10.559056 7070 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:57:10.559226 7070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/master-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/master-0/wal
uuid: "73109dfa9b4d4463a096a15f0501a28c"
format_stamp: "Formatted at 2026-09-28 14:57:10 on dist-test-slave-j968"
server_key: "dc5b76cff65eb2db69292276f53d7a23"
server_key_iv: "0aca73e9ab7105923276edac30d6057a"
server_key_version: "encryptionkey@0"
I20260928 14:57:10.559437 7070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:10.584141 7070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:10.588701 7070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:10.588953 7070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:10.612439 7070 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.62:32977
I20260928 14:57:10.612478 7146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.62:32977 every 8 connection(s)
I20260928 14:57:10.613873 7070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
I20260928 14:57:10.617856 7147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:10.623953 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 7070
I20260928 14:57:10.624181 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/master-0/wal/instance
I20260928 14:57:10.625029 31904 external_mini_cluster.cc:1428] Setting key f6715ce5dc7498f14303085cdf175009
I20260928 14:57:10.631501 7147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73109dfa9b4d4463a096a15f0501a28c: Bootstrap starting.
I20260928 14:57:10.635432 7147 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73109dfa9b4d4463a096a15f0501a28c: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:10.636716 7147 log.cc:824] T 00000000000000000000000000000000 P 73109dfa9b4d4463a096a15f0501a28c: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:10.640466 7147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73109dfa9b4d4463a096a15f0501a28c: No bootstrap required, opened a new log
Sep 28 14:57:10 dist-test-slave-j968 krb5kdc[7057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM
I20260928 14:57:10.651691 7147 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73109dfa9b4d4463a096a15f0501a28c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73109dfa9b4d4463a096a15f0501a28c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32977 } }
I20260928 14:57:10.652230 7147 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73109dfa9b4d4463a096a15f0501a28c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:10.652364 7150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:10.627193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42420 (local address 127.31.40.62:32977)
0928 14:57:10.628065 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:10.628081 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:10.628126 (+ 45us) server_negotiation.cc:407] Connection header received
0928 14:57:10.629860 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:10.629886 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:10.630308 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:10.630688 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:10.632535 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:10.633996 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:10.635860 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:10.636693 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:10.636893 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:10.641944 (+ 5051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:10.641957 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:10.641982 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:10.642037 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:10.646779 (+ 4742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:10.647733 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:10.647743 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:10.647755 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:10.648019 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:10.648576 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:10.648581 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:10.648585 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:10.649455 (+ 870us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:10.649784 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:10.650244 (+ 460us) server_negotiation.cc:299] Negotiation successful
0928 14:57:10.650863 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":402,"threads_started":1}
I20260928 14:57:10.652366 7147 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73109dfa9b4d4463a096a15f0501a28c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73109dfa9b4d4463a096a15f0501a28c, State: Initialized, Role: FOLLOWER
I20260928 14:57:10.653167 7147 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73109dfa9b4d4463a096a15f0501a28c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73109dfa9b4d4463a096a15f0501a28c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32977 } }
I20260928 14:57:10.653430 7147 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73109dfa9b4d4463a096a15f0501a28c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:57:10.653589 7147 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73109dfa9b4d4463a096a15f0501a28c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:57:10.653784 7147 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73109dfa9b4d4463a096a15f0501a28c [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:10.656288 7147 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73109dfa9b4d4463a096a15f0501a28c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73109dfa9b4d4463a096a15f0501a28c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32977 } }
I20260928 14:57:10.656840 7147 leader_election.cc:302] T 00000000000000000000000000000000 P 73109dfa9b4d4463a096a15f0501a28c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73109dfa9b4d4463a096a15f0501a28c; no voters:
I20260928 14:57:10.657502 7147 leader_election.cc:288] T 00000000000000000000000000000000 P 73109dfa9b4d4463a096a15f0501a28c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:57:10.657642 7152 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73109dfa9b4d4463a096a15f0501a28c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:57:10.658654 7152 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73109dfa9b4d4463a096a15f0501a28c [term 1 LEADER]: Becoming Leader. State: Replica: 73109dfa9b4d4463a096a15f0501a28c, State: Running, Role: LEADER
I20260928 14:57:10.659219 7152 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73109dfa9b4d4463a096a15f0501a28c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73109dfa9b4d4463a096a15f0501a28c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32977 } }
I20260928 14:57:10.659740 7147 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73109dfa9b4d4463a096a15f0501a28c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:57:10.661985 7154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73109dfa9b4d4463a096a15f0501a28c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73109dfa9b4d4463a096a15f0501a28c. Latest consensus state: current_term: 1 leader_uuid: "73109dfa9b4d4463a096a15f0501a28c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73109dfa9b4d4463a096a15f0501a28c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32977 } } }
I20260928 14:57:10.662400 7154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73109dfa9b4d4463a096a15f0501a28c [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:10.662312 7153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73109dfa9b4d4463a096a15f0501a28c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73109dfa9b4d4463a096a15f0501a28c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73109dfa9b4d4463a096a15f0501a28c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32977 } } }
I20260928 14:57:10.662760 7153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73109dfa9b4d4463a096a15f0501a28c [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:10.664837 7157 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:57:10.669730 7157 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:57:10.679289 7157 catalog_manager.cc:1347] Generated new cluster ID: beb2d43c76874706ad963f8522fdb395
I20260928 14:57:10.679418 7157 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:57:10.706929 7157 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:57:10.708596 7157 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:57:10.721263 7157 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73109dfa9b4d4463a096a15f0501a28c: Generated new TSK 0
I20260928 14:57:10.722579 7157 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.31.40.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.40.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.40.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/krb5kdc/oryx_127.31.40.1.keytab.
Entry for principal oryx/127.31.40.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/krb5kdc/oryx_127.31.40.1.keytab.
I20260928 14:57:10.792726 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.1:0
--local_ip_for_outbound_sockets=127.31.40.1
--webserver_interface=127.31.40.1
--webserver_port=0
--tserver_master_addrs=127.31.40.62:32977
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/krb5kdc/oryx_127.31.40.1.keytab
--principal=oryx/127.31.40.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:34277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:57:11.139179 7173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:11.139546 7173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:11.139731 7173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:11.147540 7173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:11.147722 7173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:11.147888 7173 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:57:11.148038 7173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.1
I20260928 14:57:11.158262 7173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:34277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/krb5kdc/oryx_127.31.40.1.keytab
--principal=oryx/127.31.40.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.40.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.40.62:32977
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.1
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:57:11.159746 7173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:11.161275 7173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:11.175406 7178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:11.175359 7179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:11.175315 7181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:11.176165 7173 server_base.cc:1034] running on GCE node
I20260928 14:57:11.176867 7173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:11.177821 7173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:11.179032 7173 hybrid_clock.cc:648] HybridClock initialized: now 1790607431178971 us; error 42 us; skew 500 ppm
I20260928 14:57:11.179343 7173 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:57:11 dist-test-slave-j968 krb5kdc[7057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607431, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.40.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:11.182698 7173 init.cc:377] Logged in from keytab as oryx/127.31.40.1@KRBTEST.COM (short username oryx)
I20260928 14:57:11.184325 7173 webserver.cc:460] Webserver started at http://127.31.40.1:41617/ using document root <none> and password file <none>
I20260928 14:57:11.185077 7173 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:11.185166 7173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:11.185523 7173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:11.188548 7173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-0/data/instance:
uuid: "9659e13c2242471c8899cf4f87d3fa36"
format_stamp: "Formatted at 2026-09-28 14:57:11 on dist-test-slave-j968"
server_key: "3c1759ce7da5cdaee2a2182d447833fc"
server_key_iv: "18ada4948f4ef64785686e4f729dc189"
server_key_version: "encryptionkey@0"
I20260928 14:57:11.189364 7173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-0/wal/instance:
uuid: "9659e13c2242471c8899cf4f87d3fa36"
format_stamp: "Formatted at 2026-09-28 14:57:11 on dist-test-slave-j968"
server_key: "3c1759ce7da5cdaee2a2182d447833fc"
server_key_iv: "18ada4948f4ef64785686e4f729dc189"
server_key_version: "encryptionkey@0"
I20260928 14:57:11.194782 7173 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 14:57:11.198499 7188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:11.200104 7173 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 14:57:11.200273 7173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-0/wal
uuid: "9659e13c2242471c8899cf4f87d3fa36"
format_stamp: "Formatted at 2026-09-28 14:57:11 on dist-test-slave-j968"
server_key: "3c1759ce7da5cdaee2a2182d447833fc"
server_key_iv: "18ada4948f4ef64785686e4f729dc189"
server_key_version: "encryptionkey@0"
I20260928 14:57:11.200502 7173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:11.214809 7173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:11.219518 7173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:11.219823 7173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:11.222841 7173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:57:11.222961 7173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:11.223116 7173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:57:11.223188 7173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 14:57:11 dist-test-slave-j968 krb5kdc[7057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607431, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.40.1@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM
I20260928 14:57:11.250247 7197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:11.223694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:60413 (local address 127.31.40.62:32977)
0928 14:57:11.224289 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:11.224294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:11.225543 (+ 1249us) server_negotiation.cc:407] Connection header received
0928 14:57:11.228047 (+ 2504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:11.228058 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:11.228205 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:11.228347 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:11.230535 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:11.231805 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:11.235204 (+ 3399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:11.235746 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:11.235914 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:11.241539 (+ 5625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:11.241548 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:11.241554 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:11.241613 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:11.245250 (+ 3637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:11.246667 (+ 1417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:11.246673 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:11.246676 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:11.246846 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:11.247620 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:11.247626 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:11.247630 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:11.248252 (+ 622us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:11.248439 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:11.249618 (+ 1179us) server_negotiation.cc:299] Negotiation successful
0928 14:57:11.249863 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":250,"threads_started":1}
I20260928 14:57:11.251924 7198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:11.224499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:32977 (local address 127.31.40.1:60413)
0928 14:57:11.225372 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:11.225410 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:11.227596 (+ 2186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:11.228662 (+ 1066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:11.228680 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:11.229662 (+ 982us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:11.230326 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:11.230345 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:11.232075 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:11.232085 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:11.234997 (+ 2912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:11.235023 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:11.236068 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:11.236080 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:11.236455 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:11.237441 (+ 986us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:11.237482 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:11.241265 (+ 3783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:11.245446 (+ 4181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:11.245468 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:11.245497 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:11.246416 (+ 919us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:11.247032 (+ 616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:11.247039 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:11.247045 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:11.247427 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:11.248446 (+ 1019us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:11.248460 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:11.249086 (+ 626us) client_negotiation.cc:769] Sending connection context
0928 14:57:11.249516 (+ 430us) client_negotiation.cc:240] Negotiation successful
0928 14:57:11.250138 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":383,"threads_started":1}
I20260928 14:57:11.272233 7173 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.1:43671
I20260928 14:57:11.272249 7308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.1:43671 every 8 connection(s)
I20260928 14:57:11.273617 7173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
I20260928 14:57:11.275326 7290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:11.259584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:32977 (local address 127.31.40.1:35063)
0928 14:57:11.260276 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:11.260291 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:11.260460 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:11.261050 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:11.261055 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:11.261541 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:11.261898 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:11.261909 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:11.263262 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:11.263271 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:11.265538 (+ 2267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:11.265560 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:11.266968 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:11.266973 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:11.267097 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:11.267803 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:11.267833 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:11.268955 (+ 1122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:11.272691 (+ 3736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:11.272696 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:11.272707 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:11.273096 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:11.273497 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:11.273501 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:11.273503 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:11.273627 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:11.274152 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:11.274156 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:11.274345 (+ 189us) client_negotiation.cc:769] Sending connection context
0928 14:57:11.274508 (+ 163us) client_negotiation.cc:240] Negotiation successful
0928 14:57:11.274904 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":432,"threads_started":1}
I20260928 14:57:11.275957 7309 heartbeater.cc:347] Connected to a master server at 127.31.40.62:32977
I20260928 14:57:11.276180 7197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:11.259505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:35063 (local address 127.31.40.62:32977)
0928 14:57:11.259755 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:11.259759 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:11.260383 (+ 624us) server_negotiation.cc:407] Connection header received
0928 14:57:11.260638 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:11.260645 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:11.260740 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:11.260842 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:11.262054 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:11.263049 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:11.265811 (+ 2762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:11.266763 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:11.266930 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:11.269139 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:11.269145 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:11.269149 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:11.269186 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:11.272556 (+ 3370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:11.273226 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:11.273231 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:11.273235 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:11.273368 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:11.273741 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:11.273746 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:11.273751 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:11.274025 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:11.274187 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:11.275653 (+ 1466us) server_negotiation.cc:299] Negotiation successful
0928 14:57:11.275845 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 14:57:11.276242 7309 heartbeater.cc:464] Registering TS with master...
I20260928 14:57:11.277064 7309 heartbeater.cc:511] Master 127.31.40.62:32977 requested a full tablet report, sending...
I20260928 14:57:11.278435 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 7173
I20260928 14:57:11.278636 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-0/wal/instance
I20260928 14:57:11.279466 31904 external_mini_cluster.cc:1428] Setting key 163d73e4578fe784c88832076e5219d6
I20260928 14:57:11.279444 7102 ts_manager.cc:194] Registered new tserver with Master: 9659e13c2242471c8899cf4f87d3fa36 (127.31.40.1:43671)
I20260928 14:57:11.281424 7102 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.40.1@KRBTEST.COM'} at 127.31.40.1:60413
WARNING: no policy specified for oryx/127.31.40.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.40.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.40.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/krb5kdc/oryx_127.31.40.2.keytab.
Entry for principal oryx/127.31.40.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/krb5kdc/oryx_127.31.40.2.keytab.
I20260928 14:57:11.329694 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.2:0
--local_ip_for_outbound_sockets=127.31.40.2
--webserver_interface=127.31.40.2
--webserver_port=0
--tserver_master_addrs=127.31.40.62:32977
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/krb5kdc/oryx_127.31.40.2.keytab
--principal=oryx/127.31.40.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:34277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:57:11.675697 7313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:11.676054 7313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:11.676240 7313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:11.684497 7313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:11.684633 7313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:11.684809 7313 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:57:11.684960 7313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.2
I20260928 14:57:11.696209 7313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:34277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/krb5kdc/oryx_127.31.40.2.keytab
--principal=oryx/127.31.40.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.40.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.40.62:32977
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.2
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:57:11.697996 7313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:11.699741 7313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:11.714262 7318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:11.714325 7319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:11.714354 7321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:11.715488 7313 server_base.cc:1034] running on GCE node
I20260928 14:57:11.716193 7313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:11.717171 7313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:11.718405 7313 hybrid_clock.cc:648] HybridClock initialized: now 1790607431718338 us; error 35 us; skew 500 ppm
I20260928 14:57:11.718711 7313 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:57:11 dist-test-slave-j968 krb5kdc[7057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607431, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.40.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:11.722496 7313 init.cc:377] Logged in from keytab as oryx/127.31.40.2@KRBTEST.COM (short username oryx)
I20260928 14:57:11.724293 7313 webserver.cc:460] Webserver started at http://127.31.40.2:45867/ using document root <none> and password file <none>
I20260928 14:57:11.725059 7313 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:11.725152 7313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:11.725453 7313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:11.728576 7313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-1/data/instance:
uuid: "48c7f448d8ae483ca98628d00e3b6fcc"
format_stamp: "Formatted at 2026-09-28 14:57:11 on dist-test-slave-j968"
server_key: "fb1b088feb44635435b71cecc2b35364"
server_key_iv: "34502257ced226be4a600a06167c1548"
server_key_version: "encryptionkey@0"
I20260928 14:57:11.729316 7313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-1/wal/instance:
uuid: "48c7f448d8ae483ca98628d00e3b6fcc"
format_stamp: "Formatted at 2026-09-28 14:57:11 on dist-test-slave-j968"
server_key: "fb1b088feb44635435b71cecc2b35364"
server_key_iv: "34502257ced226be4a600a06167c1548"
server_key_version: "encryptionkey@0"
I20260928 14:57:11.734695 7313 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 14:57:11.738512 7328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:11.740165 7313 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:57:11.740319 7313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-1/wal
uuid: "48c7f448d8ae483ca98628d00e3b6fcc"
format_stamp: "Formatted at 2026-09-28 14:57:11 on dist-test-slave-j968"
server_key: "fb1b088feb44635435b71cecc2b35364"
server_key_iv: "34502257ced226be4a600a06167c1548"
server_key_version: "encryptionkey@0"
I20260928 14:57:11.740525 7313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:11.774735 7313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:11.780156 7313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:11.780486 7313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:11.784727 7313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:57:11.784967 7313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 14:57:11.785198 7313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:57:11.785326 7313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 14:57:11 dist-test-slave-j968 krb5kdc[7057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607431, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.40.2@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM
I20260928 14:57:11.817941 7336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:11.785082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:49207 (local address 127.31.40.62:32977)
0928 14:57:11.793512 (+ 8430us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:11.793520 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:11.793543 (+ 23us) server_negotiation.cc:407] Connection header received
0928 14:57:11.793626 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:11.793633 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:11.793746 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:11.793876 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:11.796798 (+ 2922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:11.797774 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:11.801391 (+ 3617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:11.801955 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:11.802116 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:11.808779 (+ 6663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:11.808789 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:11.808796 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:11.808843 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:11.813118 (+ 4275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:11.814631 (+ 1513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:11.814638 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:11.814644 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:11.814813 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:11.815500 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:11.815510 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:11.815516 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:11.816021 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:11.816227 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:11.817304 (+ 1077us) server_negotiation.cc:299] Negotiation successful
0928 14:57:11.817544 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8290,"thread_start_us":8185,"threads_started":1}
I20260928 14:57:11.819480 7338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:11.785895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:32977 (local address 127.31.40.2:49207)
0928 14:57:11.790426 (+ 4531us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:11.790490 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:11.793040 (+ 2550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:11.794198 (+ 1158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:11.794210 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:11.795266 (+ 1056us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:11.796524 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:11.796562 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:11.797965 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:11.797971 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:11.801155 (+ 3184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:11.801196 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:11.802256 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:11.802268 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:11.802723 (+ 455us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:11.803893 (+ 1170us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:11.803945 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:11.808392 (+ 4447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:11.813349 (+ 4957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:11.813362 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:11.813403 (+ 41us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:11.814356 (+ 953us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:11.814945 (+ 589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:11.814950 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:11.814955 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:11.815285 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:11.816192 (+ 907us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:11.816205 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:11.816791 (+ 586us) client_negotiation.cc:769] Sending connection context
0928 14:57:11.817196 (+ 405us) client_negotiation.cc:240] Negotiation successful
0928 14:57:11.817831 (+ 635us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4204,"thread_start_us":3939,"threads_started":1}
I20260928 14:57:11.835386 7313 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.2:34247
I20260928 14:57:11.835403 7448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.2:34247 every 8 connection(s)
I20260928 14:57:11.837110 7313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
I20260928 14:57:11.839450 7449 heartbeater.cc:347] Connected to a master server at 127.31.40.62:32977
I20260928 14:57:11.839702 7449 heartbeater.cc:464] Registering TS with master...
I20260928 14:57:11.840499 7449 heartbeater.cc:511] Master 127.31.40.62:32977 requested a full tablet report, sending...
I20260928 14:57:11.842144 7102 ts_manager.cc:194] Registered new tserver with Master: 48c7f448d8ae483ca98628d00e3b6fcc (127.31.40.2:34247)
I20260928 14:57:11.843061 7429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:11.827367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:32977 (local address 127.31.40.2:43345)
0928 14:57:11.828280 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:11.828309 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:11.828507 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:11.829092 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:11.829099 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:11.829792 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:11.830448 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:11.830464 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:11.831522 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:11.831527 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:11.833228 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:11.833243 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:11.834148 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:11.834157 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:11.834323 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:11.835051 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:11.835089 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:11.836333 (+ 1244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:11.840065 (+ 3732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:11.840069 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:11.840072 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:11.840511 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:11.840957 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:11.840960 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:11.841010 (+ 50us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:11.841136 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:11.841879 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:11.841884 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:11.842156 (+ 272us) client_negotiation.cc:769] Sending connection context
0928 14:57:11.842302 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 14:57:11.842615 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":761,"thread_start_us":608,"threads_started":1}
I20260928 14:57:11.843277 7102 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.40.2@KRBTEST.COM'} at 127.31.40.2:49207
I20260928 14:57:11.844059 7336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:11.827112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:43345 (local address 127.31.40.62:32977)
0928 14:57:11.827321 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:11.827328 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:11.828435 (+ 1107us) server_negotiation.cc:407] Connection header received
0928 14:57:11.828663 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:11.828671 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:11.828775 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:11.828880 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:11.830628 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:11.831368 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:11.833401 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:11.833913 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:11.834063 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:11.836537 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:11.836542 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:11.836545 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:11.836586 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:11.839821 (+ 3235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:11.840682 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:11.840689 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:11.840695 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:11.840841 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:11.841263 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:11.841268 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:11.841271 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:11.841739 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:11.841877 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:11.843370 (+ 1493us) server_negotiation.cc:299] Negotiation successful
0928 14:57:11.843618 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 14:57:11.844434 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 7313
I20260928 14:57:11.844623 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-1/wal/instance
I20260928 14:57:11.845201 31904 external_mini_cluster.cc:1428] Setting key d13122a5c16e497e1f9d36c6e899794e
WARNING: no policy specified for oryx/127.31.40.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.40.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.40.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/krb5kdc/oryx_127.31.40.3.keytab.
Entry for principal oryx/127.31.40.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/krb5kdc/oryx_127.31.40.3.keytab.
I20260928 14:57:11.895741 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.3:0
--local_ip_for_outbound_sockets=127.31.40.3
--webserver_interface=127.31.40.3
--webserver_port=0
--tserver_master_addrs=127.31.40.62:32977
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/krb5kdc/oryx_127.31.40.3.keytab
--principal=oryx/127.31.40.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:34277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:57:12.238513 7453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:12.238814 7453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:12.238999 7453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:12.246637 7453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:12.246758 7453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:12.246906 7453 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:57:12.247030 7453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.3
I20260928 14:57:12.257414 7453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:34277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/krb5kdc/oryx_127.31.40.3.keytab
--principal=oryx/127.31.40.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.40.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.40.62:32977
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.3
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:57:12.258811 7453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:12.260322 7453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:12.274241 7458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:12.274212 7459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:12.275197 7461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:12.275583 7453 server_base.cc:1034] running on GCE node
I20260928 14:57:12.276292 7453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:12.277362 7453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:12.278636 7453 hybrid_clock.cc:648] HybridClock initialized: now 1790607432278569 us; error 42 us; skew 500 ppm
I20260928 14:57:12.278975 7453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:57:12 dist-test-slave-j968 krb5kdc[7057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607432, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.40.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:12.282325 7453 init.cc:377] Logged in from keytab as oryx/127.31.40.3@KRBTEST.COM (short username oryx)
I20260928 14:57:12.283821 7453 webserver.cc:460] Webserver started at http://127.31.40.3:35815/ using document root <none> and password file <none>
I20260928 14:57:12.284579 7453 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:12.284794 7453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:12.284896 7309 heartbeater.cc:503] Master 127.31.40.62:32977 was elected leader, sending a full tablet report...
I20260928 14:57:12.285183 7453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:12.288717 7453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-2/data/instance:
uuid: "2a617e1e19a24cbc8c99e83995afc049"
format_stamp: "Formatted at 2026-09-28 14:57:12 on dist-test-slave-j968"
server_key: "cf1cdb845ccfef87853f5c190decf0ec"
server_key_iv: "28aa1c537a917293dd9b405a6d2a2797"
server_key_version: "encryptionkey@0"
I20260928 14:57:12.289460 7453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-2/wal/instance:
uuid: "2a617e1e19a24cbc8c99e83995afc049"
format_stamp: "Formatted at 2026-09-28 14:57:12 on dist-test-slave-j968"
server_key: "cf1cdb845ccfef87853f5c190decf0ec"
server_key_iv: "28aa1c537a917293dd9b405a6d2a2797"
server_key_version: "encryptionkey@0"
I20260928 14:57:12.294672 7453 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 14:57:12.298305 7468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:12.299763 7453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 14:57:12.299980 7453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-2/wal
uuid: "2a617e1e19a24cbc8c99e83995afc049"
format_stamp: "Formatted at 2026-09-28 14:57:12 on dist-test-slave-j968"
server_key: "cf1cdb845ccfef87853f5c190decf0ec"
server_key_iv: "28aa1c537a917293dd9b405a6d2a2797"
server_key_version: "encryptionkey@0"
I20260928 14:57:12.300205 7453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:12.318128 7453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:12.323019 7453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:12.323367 7453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:12.326169 7453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:57:12.326252 7453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:12.326429 7453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:57:12.326503 7453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 14:57:12 dist-test-slave-j968 krb5kdc[7057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607432, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.40.3@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM
I20260928 14:57:12.352830 7336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:12.327262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:43559 (local address 127.31.40.62:32977)
0928 14:57:12.327450 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:12.327455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:12.329264 (+ 1809us) server_negotiation.cc:407] Connection header received
0928 14:57:12.331662 (+ 2398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:12.331672 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:12.331761 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:12.331869 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:12.334824 (+ 2955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:12.335589 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:12.338467 (+ 2878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:12.339126 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:12.339311 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:12.345310 (+ 5999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:12.345319 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:12.345345 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:12.345405 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:12.349235 (+ 3830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:12.350158 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:12.350163 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:12.350165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:12.350304 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:12.350827 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:12.350835 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:12.350840 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:12.351171 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:12.351329 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:12.352205 (+ 876us) server_negotiation.cc:299] Negotiation successful
0928 14:57:12.352450 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 14:57:12.354568 7477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:12.327979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:32977 (local address 127.31.40.3:43559)
0928 14:57:12.329059 (+ 1080us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:12.329115 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:12.331225 (+ 2110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:12.332550 (+ 1325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:12.332572 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:12.333526 (+ 954us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:12.334617 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:12.334641 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:12.335781 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:12.335789 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:12.338290 (+ 2501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:12.338304 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:12.339488 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:12.339503 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:12.339954 (+ 451us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:12.340956 (+ 1002us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:12.341000 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:12.344935 (+ 3935us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:12.349417 (+ 4482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:12.349429 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:12.349449 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:12.349998 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:12.350424 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:12.350429 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:12.350433 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:12.350678 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:12.351305 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:12.351315 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:12.351733 (+ 418us) client_negotiation.cc:769] Sending connection context
0928 14:57:12.352093 (+ 360us) client_negotiation.cc:240] Negotiation successful
0928 14:57:12.352809 (+ 716us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"thread_start_us":422,"threads_started":1}
I20260928 14:57:12.377519 7336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:12.361938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:58335 (local address 127.31.40.62:32977)
0928 14:57:12.362190 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:12.362199 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:12.362828 (+ 629us) server_negotiation.cc:407] Connection header received
0928 14:57:12.363046 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:12.363055 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:12.363202 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:12.363365 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:12.364856 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:12.366256 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:12.367642 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:12.368476 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:12.368619 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:12.370959 (+ 2340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:12.370965 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:12.370971 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:12.371016 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:12.374492 (+ 3476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:12.375175 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:12.375182 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:12.375187 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:12.375373 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:12.375865 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:12.375869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:12.375872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:12.376264 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:12.376421 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:12.376851 (+ 430us) server_negotiation.cc:299] Negotiation successful
0928 14:57:12.377117 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 14:57:12.377651 7553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:12.361929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:32977 (local address 127.31.40.3:58335)
0928 14:57:12.362695 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:12.362714 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:12.362907 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:12.363379 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:12.363383 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:12.363951 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:12.364574 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:12.364594 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:12.366437 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:12.366449 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:12.367478 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:12.367498 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:12.368669 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:12.368674 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:12.368812 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:12.369518 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:12.369535 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:12.370741 (+ 1206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:12.374631 (+ 3890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:12.374637 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:12.374641 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:12.375046 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:12.375521 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:12.375526 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:12.375530 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:12.375672 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:12.376390 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:12.376394 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:12.376596 (+ 202us) client_negotiation.cc:769] Sending connection context
0928 14:57:12.376745 (+ 149us) client_negotiation.cc:240] Negotiation successful
0928 14:57:12.377150 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":454,"threads_started":1}
I20260928 14:57:12.383255 7453 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.3:35371
I20260928 14:57:12.383302 7587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.3:35371 every 8 connection(s)
I20260928 14:57:12.384613 7453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
I20260928 14:57:12.386782 7588 heartbeater.cc:347] Connected to a master server at 127.31.40.62:32977
I20260928 14:57:12.387094 7588 heartbeater.cc:464] Registering TS with master...
I20260928 14:57:12.387760 7588 heartbeater.cc:511] Master 127.31.40.62:32977 requested a full tablet report, sending...
I20260928 14:57:12.389096 7102 ts_manager.cc:194] Registered new tserver with Master: 2a617e1e19a24cbc8c99e83995afc049 (127.31.40.3:35371)
I20260928 14:57:12.389932 7102 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.40.3@KRBTEST.COM'} at 127.31.40.3:43559
I20260928 14:57:12.390456 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 7453
I20260928 14:57:12.390641 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-2/wal/instance
I20260928 14:57:12.391114 31904 external_mini_cluster.cc:1428] Setting key e536f1ae76e5c5adaf15763327c6dac6
I20260928 14:57:12.394271 31904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 14:57:12 dist-test-slave-j968 krb5kdc[7057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607432, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 14:57:12 dist-test-slave-j968 krb5kdc[7057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:57:12 dist-test-slave-j968 krb5kdc[7057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 14:57:12.446221 7336 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:12.435221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42432 (local address 127.31.40.62:32977)
0928 14:57:12.435421 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:12.435426 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:12.435773 (+ 347us) server_negotiation.cc:407] Connection header received
0928 14:57:12.436006 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:12.436010 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:12.436077 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:12.436173 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:12.437536 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:12.438255 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:12.439448 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:12.439996 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:12.440116 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:12.445883 (+ 5767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42432: BlockingRecv error: recv got EOF from 127.0.0.1:42432 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 28 14:57:12 dist-test-slave-j968 krb5kdc[7057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607432, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM
I20260928 14:57:12.464766 7336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:12.449209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42442 (local address 127.31.40.62:32977)
0928 14:57:12.449449 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:12.449457 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:12.449833 (+ 376us) server_negotiation.cc:407] Connection header received
0928 14:57:12.450109 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:12.450115 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:12.450201 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:12.450314 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:12.451540 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:12.452379 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:12.453679 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:12.454137 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:12.454288 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:12.458522 (+ 4234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:12.458528 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:12.458531 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:12.458569 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:12.461512 (+ 2943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:12.462581 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:12.462587 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:12.462591 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:12.462744 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:12.463136 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:12.463142 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:12.463145 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:12.463473 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:12.463748 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:12.464163 (+ 415us) server_negotiation.cc:299] Negotiation successful
0928 14:57:12.464429 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 14:57:12.469389 7102 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42442:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:57:12.476920 7102 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:57:12.501574 7606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:12.489206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:43671 (local address 127.0.0.1:51354)
0928 14:57:12.490097 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:12.490153 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:12.490389 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:12.492311 (+ 1922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:12.492321 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:12.492354 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:12.492891 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:12.492904 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:12.495077 (+ 2173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:12.495085 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:12.499041 (+ 3956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:12.499053 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:12.500488 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:12.500493 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:12.500710 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:12.500773 (+ 63us) client_negotiation.cc:769] Sending connection context
0928 14:57:12.500910 (+ 137us) client_negotiation.cc:240] Negotiation successful
0928 14:57:12.501139 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"mutex_wait_us":55,"thread_start_us":343,"threads_started":1}
I20260928 14:57:12.502631 7610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:12.490499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35654 (local address 127.31.40.2:34247)
0928 14:57:12.491064 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:12.491077 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:12.492558 (+ 1481us) server_negotiation.cc:407] Connection header received
0928 14:57:12.492882 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:12.492890 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:12.493161 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:12.493338 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:12.494202 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:12.495667 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:12.499396 (+ 3729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:12.500849 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:12.501196 (+ 347us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:12.501354 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:12.502288 (+ 934us) server_negotiation.cc:299] Negotiation successful
0928 14:57:12.502422 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":229,"threads_started":1}
I20260928 14:57:12.502693 7607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:12.489436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51354 (local address 127.31.40.1:43671)
0928 14:57:12.491510 (+ 2074us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:12.491519 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:12.491551 (+ 32us) server_negotiation.cc:407] Connection header received
0928 14:57:12.491665 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:12.491674 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:12.491983 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:12.493132 (+ 1149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:12.493361 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:12.494944 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:12.499220 (+ 4276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:12.500289 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:12.500450 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:12.500701 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:12.502211 (+ 1510us) server_negotiation.cc:299] Negotiation successful
0928 14:57:12.502411 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1901,"thread_start_us":1779,"threads_started":1}
I20260928 14:57:12.503994 7611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:12.490995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55132 (local address 127.31.40.3:35371)
0928 14:57:12.491911 (+ 916us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:12.491919 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:12.494653 (+ 2734us) server_negotiation.cc:407] Connection header received
0928 14:57:12.494764 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:12.494771 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:12.494979 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:12.495203 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:12.496011 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:12.497104 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:12.501189 (+ 4085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:12.502840 (+ 1651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:12.502990 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:12.503220 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:12.503444 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 14:57:12.503600 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":740,"thread_start_us":570,"threads_started":1}
I20260928 14:57:12.504056 7608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:12.490416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.3:35371 (local address 127.0.0.1:55132)
0928 14:57:12.494520 (+ 4104us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:12.494538 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:12.494645 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:12.495188 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:12.495195 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:12.495232 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:12.495705 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:12.495716 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:12.497409 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:12.497416 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:12.501013 (+ 3597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:12.501027 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:12.503046 (+ 2019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:12.503052 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:12.503236 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:12.503271 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 14:57:12.503378 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 14:57:12.503534 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3993,"mutex_wait_us":1249,"thread_start_us":1687,"threads_started":1}
I20260928 14:57:12.504483 7609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:12.489941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.2:34247 (local address 127.0.0.1:35654)
0928 14:57:12.492434 (+ 2493us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:12.492457 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:12.492661 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:12.493575 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:12.493580 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:12.493606 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:12.494006 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:12.494018 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:12.495837 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:12.495846 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:12.499232 (+ 3386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:12.499245 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:12.501708 (+ 2463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:12.501715 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:12.501882 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:12.501913 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 14:57:12.502042 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 14:57:12.503951 (+ 1909us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2320,"mutex_wait_us":85,"thread_start_us":4343,"threads_started":1}
I20260928 14:57:12.506775 7239 tablet_service.cc:1467] Processing CreateTablet for tablet bc2fec3dec1f4cd4af76ca07e08b7651 (DEFAULT_TABLE table=test-table [id=b548dbc1805c4826a9e880994894c886]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:57:12.507997 7239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc2fec3dec1f4cd4af76ca07e08b7651. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:12.507580 7379 tablet_service.cc:1467] Processing CreateTablet for tablet bc2fec3dec1f4cd4af76ca07e08b7651 (DEFAULT_TABLE table=test-table [id=b548dbc1805c4826a9e880994894c886]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:57:12.508947 7379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc2fec3dec1f4cd4af76ca07e08b7651. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:12.510567 7518 tablet_service.cc:1467] Processing CreateTablet for tablet bc2fec3dec1f4cd4af76ca07e08b7651 (DEFAULT_TABLE table=test-table [id=b548dbc1805c4826a9e880994894c886]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:57:12.512032 7518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc2fec3dec1f4cd4af76ca07e08b7651. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:12.520267 7612 tablet_bootstrap.cc:492] T bc2fec3dec1f4cd4af76ca07e08b7651 P 9659e13c2242471c8899cf4f87d3fa36: Bootstrap starting.
I20260928 14:57:12.523774 7612 tablet_bootstrap.cc:654] T bc2fec3dec1f4cd4af76ca07e08b7651 P 9659e13c2242471c8899cf4f87d3fa36: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:12.523895 7613 tablet_bootstrap.cc:492] T bc2fec3dec1f4cd4af76ca07e08b7651 P 48c7f448d8ae483ca98628d00e3b6fcc: Bootstrap starting.
I20260928 14:57:12.526213 7612 log.cc:824] T bc2fec3dec1f4cd4af76ca07e08b7651 P 9659e13c2242471c8899cf4f87d3fa36: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:12.527500 7613 tablet_bootstrap.cc:654] T bc2fec3dec1f4cd4af76ca07e08b7651 P 48c7f448d8ae483ca98628d00e3b6fcc: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:12.527630 7614 tablet_bootstrap.cc:492] T bc2fec3dec1f4cd4af76ca07e08b7651 P 2a617e1e19a24cbc8c99e83995afc049: Bootstrap starting.
I20260928 14:57:12.528826 7613 log.cc:824] T bc2fec3dec1f4cd4af76ca07e08b7651 P 48c7f448d8ae483ca98628d00e3b6fcc: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:12.529781 7612 tablet_bootstrap.cc:492] T bc2fec3dec1f4cd4af76ca07e08b7651 P 9659e13c2242471c8899cf4f87d3fa36: No bootstrap required, opened a new log
I20260928 14:57:12.530092 7612 ts_tablet_manager.cc:1397] T bc2fec3dec1f4cd4af76ca07e08b7651 P 9659e13c2242471c8899cf4f87d3fa36: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.001s
I20260928 14:57:12.531260 7614 tablet_bootstrap.cc:654] T bc2fec3dec1f4cd4af76ca07e08b7651 P 2a617e1e19a24cbc8c99e83995afc049: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:12.532196 7613 tablet_bootstrap.cc:492] T bc2fec3dec1f4cd4af76ca07e08b7651 P 48c7f448d8ae483ca98628d00e3b6fcc: No bootstrap required, opened a new log
I20260928 14:57:12.532397 7614 log.cc:824] T bc2fec3dec1f4cd4af76ca07e08b7651 P 2a617e1e19a24cbc8c99e83995afc049: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:12.532531 7613 ts_tablet_manager.cc:1397] T bc2fec3dec1f4cd4af76ca07e08b7651 P 48c7f448d8ae483ca98628d00e3b6fcc: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.006s
I20260928 14:57:12.535938 7614 tablet_bootstrap.cc:492] T bc2fec3dec1f4cd4af76ca07e08b7651 P 2a617e1e19a24cbc8c99e83995afc049: No bootstrap required, opened a new log
I20260928 14:57:12.536239 7614 ts_tablet_manager.cc:1397] T bc2fec3dec1f4cd4af76ca07e08b7651 P 2a617e1e19a24cbc8c99e83995afc049: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.002s
I20260928 14:57:12.541169 7612 raft_consensus.cc:348] T bc2fec3dec1f4cd4af76ca07e08b7651 P 9659e13c2242471c8899cf4f87d3fa36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 } } peers { permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 } } peers { permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 } }
I20260928 14:57:12.541703 7612 raft_consensus.cc:374] T bc2fec3dec1f4cd4af76ca07e08b7651 P 9659e13c2242471c8899cf4f87d3fa36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:12.541805 7612 raft_consensus.cc:729] T bc2fec3dec1f4cd4af76ca07e08b7651 P 9659e13c2242471c8899cf4f87d3fa36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9659e13c2242471c8899cf4f87d3fa36, State: Initialized, Role: FOLLOWER
I20260928 14:57:12.542537 7612 consensus_queue.cc:260] T bc2fec3dec1f4cd4af76ca07e08b7651 P 9659e13c2242471c8899cf4f87d3fa36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 } } peers { permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 } } peers { permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 } }
I20260928 14:57:12.543401 7613 raft_consensus.cc:348] T bc2fec3dec1f4cd4af76ca07e08b7651 P 48c7f448d8ae483ca98628d00e3b6fcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 } } peers { permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 } } peers { permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 } }
I20260928 14:57:12.543948 7613 raft_consensus.cc:374] T bc2fec3dec1f4cd4af76ca07e08b7651 P 48c7f448d8ae483ca98628d00e3b6fcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:12.544061 7613 raft_consensus.cc:729] T bc2fec3dec1f4cd4af76ca07e08b7651 P 48c7f448d8ae483ca98628d00e3b6fcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48c7f448d8ae483ca98628d00e3b6fcc, State: Initialized, Role: FOLLOWER
I20260928 14:57:12.544667 7612 ts_tablet_manager.cc:1428] T bc2fec3dec1f4cd4af76ca07e08b7651 P 9659e13c2242471c8899cf4f87d3fa36: Time spent starting tablet: real 0.014s user 0.004s sys 0.013s
I20260928 14:57:12.544809 7613 consensus_queue.cc:260] T bc2fec3dec1f4cd4af76ca07e08b7651 P 48c7f448d8ae483ca98628d00e3b6fcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 } } peers { permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 } } peers { permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 } }
I20260928 14:57:12.544939 7614 raft_consensus.cc:348] T bc2fec3dec1f4cd4af76ca07e08b7651 P 2a617e1e19a24cbc8c99e83995afc049 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 } } peers { permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 } } peers { permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 } }
I20260928 14:57:12.545387 7614 raft_consensus.cc:374] T bc2fec3dec1f4cd4af76ca07e08b7651 P 2a617e1e19a24cbc8c99e83995afc049 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:12.545511 7614 raft_consensus.cc:729] T bc2fec3dec1f4cd4af76ca07e08b7651 P 2a617e1e19a24cbc8c99e83995afc049 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a617e1e19a24cbc8c99e83995afc049, State: Initialized, Role: FOLLOWER
I20260928 14:57:12.546348 7449 heartbeater.cc:503] Master 127.31.40.62:32977 was elected leader, sending a full tablet report...
I20260928 14:57:12.547066 7613 ts_tablet_manager.cc:1428] T bc2fec3dec1f4cd4af76ca07e08b7651 P 48c7f448d8ae483ca98628d00e3b6fcc: Time spent starting tablet: real 0.014s user 0.009s sys 0.005s
I20260928 14:57:12.547113 7614 consensus_queue.cc:260] T bc2fec3dec1f4cd4af76ca07e08b7651 P 2a617e1e19a24cbc8c99e83995afc049 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 } } peers { permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 } } peers { permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 } }
I20260928 14:57:12.548671 7588 heartbeater.cc:503] Master 127.31.40.62:32977 was elected leader, sending a full tablet report...
I20260928 14:57:12.549253 7614 ts_tablet_manager.cc:1428] T bc2fec3dec1f4cd4af76ca07e08b7651 P 2a617e1e19a24cbc8c99e83995afc049: Time spent starting tablet: real 0.013s user 0.008s sys 0.003s
W20260928 14:57:12.590013 7450 tablet.cc:2369] T bc2fec3dec1f4cd4af76ca07e08b7651 P 48c7f448d8ae483ca98628d00e3b6fcc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:57:12.636252 7589 tablet.cc:2369] T bc2fec3dec1f4cd4af76ca07e08b7651 P 2a617e1e19a24cbc8c99e83995afc049: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:57:12.645103 7618 raft_consensus.cc:482] T bc2fec3dec1f4cd4af76ca07e08b7651 P 9659e13c2242471c8899cf4f87d3fa36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:57:12.645551 7618 raft_consensus.cc:504] T bc2fec3dec1f4cd4af76ca07e08b7651 P 9659e13c2242471c8899cf4f87d3fa36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 } } peers { permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 } } peers { permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 } }
I20260928 14:57:12.647511 7618 leader_election.cc:288] T bc2fec3dec1f4cd4af76ca07e08b7651 P 9659e13c2242471c8899cf4f87d3fa36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a617e1e19a24cbc8c99e83995afc049 (127.31.40.3:35371), 48c7f448d8ae483ca98628d00e3b6fcc (127.31.40.2:34247)
I20260928 14:57:12.654806 7610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:12.648143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:43305 (local address 127.31.40.2:34247)
0928 14:57:12.648372 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:12.648380 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:12.648880 (+ 500us) server_negotiation.cc:407] Connection header received
0928 14:57:12.649072 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:12.649076 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:12.649134 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:12.649237 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:12.649848 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:12.651033 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:12.653008 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:12.653800 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:12.653966 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:12.654084 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:12.654393 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 14:57:12.654529 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 14:57:12.654790 7611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:12.647821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:49453 (local address 127.31.40.3:35371)
0928 14:57:12.648092 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:12.648101 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:12.648981 (+ 880us) server_negotiation.cc:407] Connection header received
0928 14:57:12.649171 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:12.649175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:12.649231 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:12.649328 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:12.649985 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:12.651033 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:12.652998 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:12.653792 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:12.653949 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:12.654079 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:12.654363 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 14:57:12.654493 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260928 14:57:12.654963 7622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:12.647779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.2:34247 (local address 127.31.40.1:43305)
0928 14:57:12.648737 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:12.648762 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:12.648913 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:12.649252 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:12.649256 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:12.649278 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:12.649682 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:12.649695 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:12.651225 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:12.651229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:12.652847 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:12.652859 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:12.654071 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:12.654077 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:12.654238 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:12.654261 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 14:57:12.654365 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 14:57:12.654495 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":814,"mutex_wait_us":27,"thread_start_us":810,"threads_started":1}
I20260928 14:57:12.655380 7621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:12.647896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.3:35371 (local address 127.31.40.1:49453)
0928 14:57:12.648865 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:12.648884 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:12.649024 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:12.649345 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:12.649349 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:12.649390 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:12.649814 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:12.649826 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:12.651174 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:12.651178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:12.652841 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:12.652852 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:12.654025 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:12.654032 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:12.654199 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:12.654224 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 14:57:12.654327 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 14:57:12.654466 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":831,"mutex_wait_us":47,"thread_start_us":400,"threads_started":1}
I20260928 14:57:12.656108 7399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc2fec3dec1f4cd4af76ca07e08b7651" candidate_uuid: "9659e13c2242471c8899cf4f87d3fa36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" is_pre_election: true
I20260928 14:57:12.656510 7538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc2fec3dec1f4cd4af76ca07e08b7651" candidate_uuid: "9659e13c2242471c8899cf4f87d3fa36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a617e1e19a24cbc8c99e83995afc049" is_pre_election: true
I20260928 14:57:12.656762 7399 raft_consensus.cc:2454] T bc2fec3dec1f4cd4af76ca07e08b7651 P 48c7f448d8ae483ca98628d00e3b6fcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9659e13c2242471c8899cf4f87d3fa36 in term 0.
I20260928 14:57:12.657083 7538 raft_consensus.cc:2454] T bc2fec3dec1f4cd4af76ca07e08b7651 P 2a617e1e19a24cbc8c99e83995afc049 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9659e13c2242471c8899cf4f87d3fa36 in term 0.
I20260928 14:57:12.657264 7619 raft_consensus.cc:482] T bc2fec3dec1f4cd4af76ca07e08b7651 P 48c7f448d8ae483ca98628d00e3b6fcc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:57:12.657572 7190 leader_election.cc:302] T bc2fec3dec1f4cd4af76ca07e08b7651 P 9659e13c2242471c8899cf4f87d3fa36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48c7f448d8ae483ca98628d00e3b6fcc, 9659e13c2242471c8899cf4f87d3fa36; no voters:
I20260928 14:57:12.657482 7619 raft_consensus.cc:504] T bc2fec3dec1f4cd4af76ca07e08b7651 P 48c7f448d8ae483ca98628d00e3b6fcc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 } } peers { permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 } } peers { permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 } }
I20260928 14:57:12.658039 7618 raft_consensus.cc:2781] T bc2fec3dec1f4cd4af76ca07e08b7651 P 9659e13c2242471c8899cf4f87d3fa36 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:57:12.658197 7618 raft_consensus.cc:482] T bc2fec3dec1f4cd4af76ca07e08b7651 P 9659e13c2242471c8899cf4f87d3fa36 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:57:12.658315 7618 raft_consensus.cc:3037] T bc2fec3dec1f4cd4af76ca07e08b7651 P 9659e13c2242471c8899cf4f87d3fa36 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:12.659142 7619 leader_election.cc:288] T bc2fec3dec1f4cd4af76ca07e08b7651 P 48c7f448d8ae483ca98628d00e3b6fcc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9659e13c2242471c8899cf4f87d3fa36 (127.31.40.1:43671), 2a617e1e19a24cbc8c99e83995afc049 (127.31.40.3:35371)
I20260928 14:57:12.662021 7618 raft_consensus.cc:504] T bc2fec3dec1f4cd4af76ca07e08b7651 P 9659e13c2242471c8899cf4f87d3fa36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 } } peers { permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 } } peers { permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 } }
I20260928 14:57:12.663033 7618 leader_election.cc:288] T bc2fec3dec1f4cd4af76ca07e08b7651 P 9659e13c2242471c8899cf4f87d3fa36 [CANDIDATE]: Term 1 election: Requested vote from peers 2a617e1e19a24cbc8c99e83995afc049 (127.31.40.3:35371), 48c7f448d8ae483ca98628d00e3b6fcc (127.31.40.2:34247)
I20260928 14:57:12.663448 7538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc2fec3dec1f4cd4af76ca07e08b7651" candidate_uuid: "9659e13c2242471c8899cf4f87d3fa36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a617e1e19a24cbc8c99e83995afc049"
I20260928 14:57:12.663733 7538 raft_consensus.cc:3037] T bc2fec3dec1f4cd4af76ca07e08b7651 P 2a617e1e19a24cbc8c99e83995afc049 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:12.666884 7607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:12.659989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:58993 (local address 127.31.40.1:43671)
0928 14:57:12.660177 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:12.660183 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:12.660355 (+ 172us) server_negotiation.cc:407] Connection header received
0928 14:57:12.660951 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:12.660959 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:12.661033 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:12.661088 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:12.661673 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:12.662535 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:12.665034 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:12.665880 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:12.666041 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:12.666156 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:12.666500 (+ 344us) server_negotiation.cc:299] Negotiation successful
0928 14:57:12.666639 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 14:57:12.666977 7623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:12.659520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:43671 (local address 127.31.40.2:58993)
0928 14:57:12.660251 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:12.660272 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:12.660426 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:12.661233 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:12.661237 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:12.661256 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:12.661541 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:12.661548 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:12.662667 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:12.662674 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:12.664866 (+ 2192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:12.664879 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:12.666186 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:12.666195 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:12.666342 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:12.666368 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 14:57:12.666475 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 14:57:12.666616 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":341,"threads_started":1}
I20260928 14:57:12.668423 7538 raft_consensus.cc:2454] T bc2fec3dec1f4cd4af76ca07e08b7651 P 2a617e1e19a24cbc8c99e83995afc049 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9659e13c2242471c8899cf4f87d3fa36 in term 1.
I20260928 14:57:12.669101 7189 leader_election.cc:302] T bc2fec3dec1f4cd4af76ca07e08b7651 P 9659e13c2242471c8899cf4f87d3fa36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a617e1e19a24cbc8c99e83995afc049, 9659e13c2242471c8899cf4f87d3fa36; no voters:
I20260928 14:57:12.669750 7399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc2fec3dec1f4cd4af76ca07e08b7651" candidate_uuid: "9659e13c2242471c8899cf4f87d3fa36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48c7f448d8ae483ca98628d00e3b6fcc"
I20260928 14:57:12.670002 7399 raft_consensus.cc:3037] T bc2fec3dec1f4cd4af76ca07e08b7651 P 48c7f448d8ae483ca98628d00e3b6fcc [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:12.671300 7259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc2fec3dec1f4cd4af76ca07e08b7651" candidate_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9659e13c2242471c8899cf4f87d3fa36" is_pre_election: true
I20260928 14:57:12.671832 7259 raft_consensus.cc:2379] T bc2fec3dec1f4cd4af76ca07e08b7651 P 9659e13c2242471c8899cf4f87d3fa36 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48c7f448d8ae483ca98628d00e3b6fcc in current term 1: Already voted for candidate 9659e13c2242471c8899cf4f87d3fa36 in this term.
I20260928 14:57:12.673288 7399 raft_consensus.cc:2454] T bc2fec3dec1f4cd4af76ca07e08b7651 P 48c7f448d8ae483ca98628d00e3b6fcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9659e13c2242471c8899cf4f87d3fa36 in term 1.
I20260928 14:57:12.673691 7611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:12.660769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:57627 (local address 127.31.40.3:35371)
0928 14:57:12.660935 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:12.660941 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:12.667337 (+ 6396us) server_negotiation.cc:407] Connection header received
0928 14:57:12.667607 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:12.667611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:12.667690 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:12.667790 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:12.668184 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:12.669240 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:12.671513 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:12.672734 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:12.672838 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:12.672923 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:12.673331 (+ 408us) server_negotiation.cc:299] Negotiation successful
0928 14:57:12.673470 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 14:57:12.673826 7623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:12.660320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.3:35371 (local address 127.31.40.2:57627)
0928 14:57:12.667240 (+ 6920us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:12.667264 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:12.667458 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:12.667805 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:12.667810 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:12.667837 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:12.668048 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:12.668056 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:12.669403 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:12.669407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:12.671327 (+ 1920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:12.671346 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:12.672881 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:12.672891 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:12.673122 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:12.673168 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 14:57:12.673288 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 14:57:12.673556 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6791,"thread_start_us":10095,"threads_started":1}
I20260928 14:57:12.674047 7618 raft_consensus.cc:2781] T bc2fec3dec1f4cd4af76ca07e08b7651 P 9659e13c2242471c8899cf4f87d3fa36 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:57:12.674446 7538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc2fec3dec1f4cd4af76ca07e08b7651" candidate_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a617e1e19a24cbc8c99e83995afc049" is_pre_election: true
I20260928 14:57:12.674595 7618 raft_consensus.cc:686] T bc2fec3dec1f4cd4af76ca07e08b7651 P 9659e13c2242471c8899cf4f87d3fa36 [term 1 LEADER]: Becoming Leader. State: Replica: 9659e13c2242471c8899cf4f87d3fa36, State: Running, Role: LEADER
I20260928 14:57:12.674811 7538 raft_consensus.cc:2379] T bc2fec3dec1f4cd4af76ca07e08b7651 P 2a617e1e19a24cbc8c99e83995afc049 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48c7f448d8ae483ca98628d00e3b6fcc in current term 1: Already voted for candidate 9659e13c2242471c8899cf4f87d3fa36 in this term.
I20260928 14:57:12.675187 7618 consensus_queue.cc:237] T bc2fec3dec1f4cd4af76ca07e08b7651 P 9659e13c2242471c8899cf4f87d3fa36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 } } peers { permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 } } peers { permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 } }
I20260928 14:57:12.675498 7329 leader_election.cc:302] T bc2fec3dec1f4cd4af76ca07e08b7651 P 48c7f448d8ae483ca98628d00e3b6fcc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48c7f448d8ae483ca98628d00e3b6fcc; no voters: 2a617e1e19a24cbc8c99e83995afc049, 9659e13c2242471c8899cf4f87d3fa36
I20260928 14:57:12.675889 7619 raft_consensus.cc:2726] T bc2fec3dec1f4cd4af76ca07e08b7651 P 48c7f448d8ae483ca98628d00e3b6fcc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 14:57:12.680651 7102 catalog_manager.cc:5495] T bc2fec3dec1f4cd4af76ca07e08b7651 P 9659e13c2242471c8899cf4f87d3fa36 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9659e13c2242471c8899cf4f87d3fa36 (127.31.40.1). New cstate: current_term: 1 leader_uuid: "9659e13c2242471c8899cf4f87d3fa36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 } health_report { overall_health: UNKNOWN } } }
I20260928 14:57:12.700608 7607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:12.695293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51368 (local address 127.31.40.1:43671)
0928 14:57:12.695490 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:12.695496 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:12.695528 (+ 32us) server_negotiation.cc:407] Connection header received
0928 14:57:12.695634 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:12.695638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:12.695728 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:12.695914 (+ 186us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:57:12.696283 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:12.697021 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:12.698313 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:12.698728 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:12.698864 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:12.699335 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:57:12.699973 (+ 638us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:57:12.700140 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:12.700197 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 14:57:12.700281 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 14:57:12.710661 7538 raft_consensus.cc:1261] T bc2fec3dec1f4cd4af76ca07e08b7651 P 2a617e1e19a24cbc8c99e83995afc049 [term 1 FOLLOWER]: Refusing update from remote peer 9659e13c2242471c8899cf4f87d3fa36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:57:12.710706 7399 raft_consensus.cc:1261] T bc2fec3dec1f4cd4af76ca07e08b7651 P 48c7f448d8ae483ca98628d00e3b6fcc [term 1 FOLLOWER]: Refusing update from remote peer 9659e13c2242471c8899cf4f87d3fa36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:57:12.711733 7618 consensus_queue.cc:1035] T bc2fec3dec1f4cd4af76ca07e08b7651 P 9659e13c2242471c8899cf4f87d3fa36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:57:12.712147 7625 consensus_queue.cc:1035] T bc2fec3dec1f4cd4af76ca07e08b7651 P 9659e13c2242471c8899cf4f87d3fa36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:57:12.724619 7628 mvcc.cc:205] Tried to move back new op lower bound from 7334328044371120128 to 7334328044242419712. Current Snapshot: MvccSnapshot[applied={T|T < 7334328044371120128}]
I20260928 14:57:12.729281 7630 mvcc.cc:205] Tried to move back new op lower bound from 7334328044371120128 to 7334328044242419712. Current Snapshot: MvccSnapshot[applied={T|T < 7334328044371120128}]
I20260928 14:57:12.743081 7101 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:57:12.743453 7101 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:57:12.748229 7101 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 73109dfa9b4d4463a096a15f0501a28c: Sending DeleteTablet for 3 replicas of tablet bc2fec3dec1f4cd4af76ca07e08b7651
I20260928 14:57:12.749678 7379 tablet_service.cc:1514] Processing DeleteTablet for tablet bc2fec3dec1f4cd4af76ca07e08b7651 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:57:12 UTC) from {username='oryx', principal='oryx/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:35654
I20260928 14:57:12.749709 7239 tablet_service.cc:1514] Processing DeleteTablet for tablet bc2fec3dec1f4cd4af76ca07e08b7651 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:57:12 UTC) from {username='oryx', principal='oryx/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:51354
I20260928 14:57:12.750800 7518 tablet_service.cc:1514] Processing DeleteTablet for tablet bc2fec3dec1f4cd4af76ca07e08b7651 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:57:12 UTC) from {username='oryx', principal='oryx/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:55132
I20260928 14:57:12.750932 7638 tablet_replica.cc:330] T bc2fec3dec1f4cd4af76ca07e08b7651 P 9659e13c2242471c8899cf4f87d3fa36: stopping tablet replica
I20260928 14:57:12.751772 7638 raft_consensus.cc:2229] T bc2fec3dec1f4cd4af76ca07e08b7651 P 9659e13c2242471c8899cf4f87d3fa36 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:57:12.752847 7638 raft_consensus.cc:2258] T bc2fec3dec1f4cd4af76ca07e08b7651 P 9659e13c2242471c8899cf4f87d3fa36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:57:12.753912 7637 tablet_replica.cc:330] T bc2fec3dec1f4cd4af76ca07e08b7651 P 48c7f448d8ae483ca98628d00e3b6fcc: stopping tablet replica
I20260928 14:57:12.754396 7637 raft_consensus.cc:2229] T bc2fec3dec1f4cd4af76ca07e08b7651 P 48c7f448d8ae483ca98628d00e3b6fcc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:57:12.754587 7101 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42442:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:57:12.755348 7101 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:57:12.758042 7638 ts_tablet_manager.cc:1905] T bc2fec3dec1f4cd4af76ca07e08b7651 P 9659e13c2242471c8899cf4f87d3fa36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:57:12.764108 7379 tablet_service.cc:1467] Processing CreateTablet for tablet 61518ef22470421f98bcfe471aee53f3 (DEFAULT_TABLE table=test-table [id=bedf9334a1c24927a09086a6cf7e1129]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:57:12.764750 7379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61518ef22470421f98bcfe471aee53f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:12.764602 7517 tablet_service.cc:1467] Processing CreateTablet for tablet 61518ef22470421f98bcfe471aee53f3 (DEFAULT_TABLE table=test-table [id=bedf9334a1c24927a09086a6cf7e1129]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:57:12.765229 7517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61518ef22470421f98bcfe471aee53f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:12.765774 7639 tablet_replica.cc:330] T bc2fec3dec1f4cd4af76ca07e08b7651 P 2a617e1e19a24cbc8c99e83995afc049: stopping tablet replica
I20260928 14:57:12.766532 7639 raft_consensus.cc:2229] T bc2fec3dec1f4cd4af76ca07e08b7651 P 2a617e1e19a24cbc8c99e83995afc049 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:57:12.767378 7639 raft_consensus.cc:2258] T bc2fec3dec1f4cd4af76ca07e08b7651 P 2a617e1e19a24cbc8c99e83995afc049 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:57:12.767946 7637 raft_consensus.cc:2258] T bc2fec3dec1f4cd4af76ca07e08b7651 P 48c7f448d8ae483ca98628d00e3b6fcc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:57:12.769855 7639 ts_tablet_manager.cc:1905] T bc2fec3dec1f4cd4af76ca07e08b7651 P 2a617e1e19a24cbc8c99e83995afc049: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:57:12.772049 7637 ts_tablet_manager.cc:1905] T bc2fec3dec1f4cd4af76ca07e08b7651 P 48c7f448d8ae483ca98628d00e3b6fcc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:57:12.772580 7613 tablet_bootstrap.cc:492] T 61518ef22470421f98bcfe471aee53f3 P 48c7f448d8ae483ca98628d00e3b6fcc: Bootstrap starting.
I20260928 14:57:12.773931 7239 tablet_service.cc:1467] Processing CreateTablet for tablet 61518ef22470421f98bcfe471aee53f3 (DEFAULT_TABLE table=test-table [id=bedf9334a1c24927a09086a6cf7e1129]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:57:12.774369 7613 tablet_bootstrap.cc:654] T 61518ef22470421f98bcfe471aee53f3 P 48c7f448d8ae483ca98628d00e3b6fcc: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:12.774520 7239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61518ef22470421f98bcfe471aee53f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:12.777669 7639 ts_tablet_manager.cc:1918] T bc2fec3dec1f4cd4af76ca07e08b7651 P 2a617e1e19a24cbc8c99e83995afc049: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:57:12.777851 7639 log.cc:1196] T bc2fec3dec1f4cd4af76ca07e08b7651 P 2a617e1e19a24cbc8c99e83995afc049: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-2/wal/wals/bc2fec3dec1f4cd4af76ca07e08b7651
I20260928 14:57:12.778401 7639 ts_tablet_manager.cc:1939] T bc2fec3dec1f4cd4af76ca07e08b7651 P 2a617e1e19a24cbc8c99e83995afc049: Deleting consensus metadata
I20260928 14:57:12.780158 7637 ts_tablet_manager.cc:1918] T bc2fec3dec1f4cd4af76ca07e08b7651 P 48c7f448d8ae483ca98628d00e3b6fcc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:57:12.780441 7637 log.cc:1196] T bc2fec3dec1f4cd4af76ca07e08b7651 P 48c7f448d8ae483ca98628d00e3b6fcc: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-1/wal/wals/bc2fec3dec1f4cd4af76ca07e08b7651
I20260928 14:57:12.780547 7086 catalog_manager.cc:4841] TS 2a617e1e19a24cbc8c99e83995afc049 (127.31.40.3:35371): tablet bc2fec3dec1f4cd4af76ca07e08b7651 (table test-table [id=b548dbc1805c4826a9e880994894c886]) successfully deleted
I20260928 14:57:12.781149 7637 ts_tablet_manager.cc:1939] T bc2fec3dec1f4cd4af76ca07e08b7651 P 48c7f448d8ae483ca98628d00e3b6fcc: Deleting consensus metadata
I20260928 14:57:12.783182 7612 tablet_bootstrap.cc:492] T 61518ef22470421f98bcfe471aee53f3 P 9659e13c2242471c8899cf4f87d3fa36: Bootstrap starting.
I20260928 14:57:12.783159 7087 catalog_manager.cc:4841] TS 48c7f448d8ae483ca98628d00e3b6fcc (127.31.40.2:34247): tablet bc2fec3dec1f4cd4af76ca07e08b7651 (table test-table [id=b548dbc1805c4826a9e880994894c886]) successfully deleted
I20260928 14:57:12.785140 7612 tablet_bootstrap.cc:654] T 61518ef22470421f98bcfe471aee53f3 P 9659e13c2242471c8899cf4f87d3fa36: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:12.788121 7638 ts_tablet_manager.cc:1918] T bc2fec3dec1f4cd4af76ca07e08b7651 P 9659e13c2242471c8899cf4f87d3fa36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:57:12.788345 7638 log.cc:1196] T bc2fec3dec1f4cd4af76ca07e08b7651 P 9659e13c2242471c8899cf4f87d3fa36: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-0/wal/wals/bc2fec3dec1f4cd4af76ca07e08b7651
I20260928 14:57:12.789083 7614 tablet_bootstrap.cc:492] T 61518ef22470421f98bcfe471aee53f3 P 2a617e1e19a24cbc8c99e83995afc049: Bootstrap starting.
I20260928 14:57:12.789299 7613 tablet_bootstrap.cc:492] T 61518ef22470421f98bcfe471aee53f3 P 48c7f448d8ae483ca98628d00e3b6fcc: No bootstrap required, opened a new log
I20260928 14:57:12.789458 7613 ts_tablet_manager.cc:1397] T 61518ef22470421f98bcfe471aee53f3 P 48c7f448d8ae483ca98628d00e3b6fcc: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.005s
I20260928 14:57:12.790344 7613 raft_consensus.cc:348] T 61518ef22470421f98bcfe471aee53f3 P 48c7f448d8ae483ca98628d00e3b6fcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 } } peers { permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 } } peers { permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 } }
I20260928 14:57:12.790594 7613 raft_consensus.cc:374] T 61518ef22470421f98bcfe471aee53f3 P 48c7f448d8ae483ca98628d00e3b6fcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:12.790701 7613 raft_consensus.cc:729] T 61518ef22470421f98bcfe471aee53f3 P 48c7f448d8ae483ca98628d00e3b6fcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48c7f448d8ae483ca98628d00e3b6fcc, State: Initialized, Role: FOLLOWER
I20260928 14:57:12.790778 7614 tablet_bootstrap.cc:654] T 61518ef22470421f98bcfe471aee53f3 P 2a617e1e19a24cbc8c99e83995afc049: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:12.791021 7613 consensus_queue.cc:260] T 61518ef22470421f98bcfe471aee53f3 P 48c7f448d8ae483ca98628d00e3b6fcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 } } peers { permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 } } peers { permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 } }
I20260928 14:57:12.792591 7613 ts_tablet_manager.cc:1428] T 61518ef22470421f98bcfe471aee53f3 P 48c7f448d8ae483ca98628d00e3b6fcc: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 14:57:12.794577 7612 tablet_bootstrap.cc:492] T 61518ef22470421f98bcfe471aee53f3 P 9659e13c2242471c8899cf4f87d3fa36: No bootstrap required, opened a new log
I20260928 14:57:12.794785 7612 ts_tablet_manager.cc:1397] T 61518ef22470421f98bcfe471aee53f3 P 9659e13c2242471c8899cf4f87d3fa36: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260928 14:57:12.795785 7612 raft_consensus.cc:348] T 61518ef22470421f98bcfe471aee53f3 P 9659e13c2242471c8899cf4f87d3fa36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 } } peers { permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 } } peers { permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 } }
I20260928 14:57:12.796072 7612 raft_consensus.cc:374] T 61518ef22470421f98bcfe471aee53f3 P 9659e13c2242471c8899cf4f87d3fa36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:12.796226 7612 raft_consensus.cc:729] T 61518ef22470421f98bcfe471aee53f3 P 9659e13c2242471c8899cf4f87d3fa36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9659e13c2242471c8899cf4f87d3fa36, State: Initialized, Role: FOLLOWER
I20260928 14:57:12.796975 7638 ts_tablet_manager.cc:1939] T bc2fec3dec1f4cd4af76ca07e08b7651 P 9659e13c2242471c8899cf4f87d3fa36: Deleting consensus metadata
I20260928 14:57:12.797499 7612 consensus_queue.cc:260] T 61518ef22470421f98bcfe471aee53f3 P 9659e13c2242471c8899cf4f87d3fa36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 } } peers { permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 } } peers { permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 } }
I20260928 14:57:12.798687 7614 tablet_bootstrap.cc:492] T 61518ef22470421f98bcfe471aee53f3 P 2a617e1e19a24cbc8c99e83995afc049: No bootstrap required, opened a new log
I20260928 14:57:12.798970 7612 ts_tablet_manager.cc:1428] T 61518ef22470421f98bcfe471aee53f3 P 9659e13c2242471c8899cf4f87d3fa36: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260928 14:57:12.799197 7614 ts_tablet_manager.cc:1397] T 61518ef22470421f98bcfe471aee53f3 P 2a617e1e19a24cbc8c99e83995afc049: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.004s
I20260928 14:57:12.799968 7086 catalog_manager.cc:4841] TS 9659e13c2242471c8899cf4f87d3fa36 (127.31.40.1:43671): tablet bc2fec3dec1f4cd4af76ca07e08b7651 (table test-table [id=b548dbc1805c4826a9e880994894c886]) successfully deleted
I20260928 14:57:12.800443 7614 raft_consensus.cc:348] T 61518ef22470421f98bcfe471aee53f3 P 2a617e1e19a24cbc8c99e83995afc049 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 } } peers { permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 } } peers { permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 } }
I20260928 14:57:12.800757 7614 raft_consensus.cc:374] T 61518ef22470421f98bcfe471aee53f3 P 2a617e1e19a24cbc8c99e83995afc049 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:12.800895 7614 raft_consensus.cc:729] T 61518ef22470421f98bcfe471aee53f3 P 2a617e1e19a24cbc8c99e83995afc049 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a617e1e19a24cbc8c99e83995afc049, State: Initialized, Role: FOLLOWER
I20260928 14:57:12.801224 7614 consensus_queue.cc:260] T 61518ef22470421f98bcfe471aee53f3 P 2a617e1e19a24cbc8c99e83995afc049 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 } } peers { permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 } } peers { permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 } }
I20260928 14:57:12.801932 7614 ts_tablet_manager.cc:1428] T 61518ef22470421f98bcfe471aee53f3 P 2a617e1e19a24cbc8c99e83995afc049: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 14:57:12.920364 7618 raft_consensus.cc:482] T 61518ef22470421f98bcfe471aee53f3 P 9659e13c2242471c8899cf4f87d3fa36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:57:12.920639 7618 raft_consensus.cc:504] T 61518ef22470421f98bcfe471aee53f3 P 9659e13c2242471c8899cf4f87d3fa36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 } } peers { permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 } } peers { permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 } }
I20260928 14:57:12.921474 7618 leader_election.cc:288] T 61518ef22470421f98bcfe471aee53f3 P 9659e13c2242471c8899cf4f87d3fa36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 48c7f448d8ae483ca98628d00e3b6fcc (127.31.40.2:34247), 2a617e1e19a24cbc8c99e83995afc049 (127.31.40.3:35371)
I20260928 14:57:12.921962 7399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61518ef22470421f98bcfe471aee53f3" candidate_uuid: "9659e13c2242471c8899cf4f87d3fa36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" is_pre_election: true
I20260928 14:57:12.922246 7399 raft_consensus.cc:2454] T 61518ef22470421f98bcfe471aee53f3 P 48c7f448d8ae483ca98628d00e3b6fcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9659e13c2242471c8899cf4f87d3fa36 in term 0.
I20260928 14:57:12.922333 7538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61518ef22470421f98bcfe471aee53f3" candidate_uuid: "9659e13c2242471c8899cf4f87d3fa36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a617e1e19a24cbc8c99e83995afc049" is_pre_election: true
I20260928 14:57:12.922667 7190 leader_election.cc:302] T 61518ef22470421f98bcfe471aee53f3 P 9659e13c2242471c8899cf4f87d3fa36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48c7f448d8ae483ca98628d00e3b6fcc, 9659e13c2242471c8899cf4f87d3fa36; no voters:
I20260928 14:57:12.922626 7538 raft_consensus.cc:2454] T 61518ef22470421f98bcfe471aee53f3 P 2a617e1e19a24cbc8c99e83995afc049 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9659e13c2242471c8899cf4f87d3fa36 in term 0.
I20260928 14:57:12.922994 7618 raft_consensus.cc:2781] T 61518ef22470421f98bcfe471aee53f3 P 9659e13c2242471c8899cf4f87d3fa36 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:57:12.923154 7618 raft_consensus.cc:482] T 61518ef22470421f98bcfe471aee53f3 P 9659e13c2242471c8899cf4f87d3fa36 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:57:12.923271 7618 raft_consensus.cc:3037] T 61518ef22470421f98bcfe471aee53f3 P 9659e13c2242471c8899cf4f87d3fa36 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:12.926476 7618 raft_consensus.cc:504] T 61518ef22470421f98bcfe471aee53f3 P 9659e13c2242471c8899cf4f87d3fa36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 } } peers { permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 } } peers { permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 } }
I20260928 14:57:12.927385 7618 leader_election.cc:288] T 61518ef22470421f98bcfe471aee53f3 P 9659e13c2242471c8899cf4f87d3fa36 [CANDIDATE]: Term 1 election: Requested vote from peers 48c7f448d8ae483ca98628d00e3b6fcc (127.31.40.2:34247), 2a617e1e19a24cbc8c99e83995afc049 (127.31.40.3:35371)
I20260928 14:57:12.928000 7538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61518ef22470421f98bcfe471aee53f3" candidate_uuid: "9659e13c2242471c8899cf4f87d3fa36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a617e1e19a24cbc8c99e83995afc049"
I20260928 14:57:12.928141 7399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61518ef22470421f98bcfe471aee53f3" candidate_uuid: "9659e13c2242471c8899cf4f87d3fa36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48c7f448d8ae483ca98628d00e3b6fcc"
I20260928 14:57:12.928390 7538 raft_consensus.cc:3037] T 61518ef22470421f98bcfe471aee53f3 P 2a617e1e19a24cbc8c99e83995afc049 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:12.928447 7399 raft_consensus.cc:3037] T 61518ef22470421f98bcfe471aee53f3 P 48c7f448d8ae483ca98628d00e3b6fcc [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:12.931905 7538 raft_consensus.cc:2454] T 61518ef22470421f98bcfe471aee53f3 P 2a617e1e19a24cbc8c99e83995afc049 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9659e13c2242471c8899cf4f87d3fa36 in term 1.
I20260928 14:57:12.931905 7399 raft_consensus.cc:2454] T 61518ef22470421f98bcfe471aee53f3 P 48c7f448d8ae483ca98628d00e3b6fcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9659e13c2242471c8899cf4f87d3fa36 in term 1.
I20260928 14:57:12.932499 7190 leader_election.cc:302] T 61518ef22470421f98bcfe471aee53f3 P 9659e13c2242471c8899cf4f87d3fa36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48c7f448d8ae483ca98628d00e3b6fcc, 9659e13c2242471c8899cf4f87d3fa36; no voters:
I20260928 14:57:12.932898 7618 raft_consensus.cc:2781] T 61518ef22470421f98bcfe471aee53f3 P 9659e13c2242471c8899cf4f87d3fa36 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:57:12.933111 7618 raft_consensus.cc:686] T 61518ef22470421f98bcfe471aee53f3 P 9659e13c2242471c8899cf4f87d3fa36 [term 1 LEADER]: Becoming Leader. State: Replica: 9659e13c2242471c8899cf4f87d3fa36, State: Running, Role: LEADER
I20260928 14:57:12.933462 7618 consensus_queue.cc:237] T 61518ef22470421f98bcfe471aee53f3 P 9659e13c2242471c8899cf4f87d3fa36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 } } peers { permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 } } peers { permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 } }
I20260928 14:57:12.936744 7102 catalog_manager.cc:5495] T 61518ef22470421f98bcfe471aee53f3 P 9659e13c2242471c8899cf4f87d3fa36 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9659e13c2242471c8899cf4f87d3fa36 (127.31.40.1). New cstate: current_term: 1 leader_uuid: "9659e13c2242471c8899cf4f87d3fa36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 } health_report { overall_health: UNKNOWN } } }
Sep 28 14:57:12 dist-test-slave-j968 krb5kdc[7057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607430, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.40.62@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM
I20260928 14:57:13.000985 7650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:12.978730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42446 (local address 127.31.40.62:32977)
0928 14:57:12.979839 (+ 1109us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:12.979843 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:12.979914 (+ 71us) server_negotiation.cc:407] Connection header received
0928 14:57:12.979979 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:12.979987 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:12.980080 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:12.980200 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:12.981865 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:12.983369 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:12.985376 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:12.985834 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:12.985958 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:12.995266 (+ 9308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:12.995271 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:12.995275 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:12.995312 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:12.998086 (+ 2774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:12.998861 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:12.998867 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:12.998871 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:12.999003 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:12.999360 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:12.999366 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:12.999370 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:12.999654 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:12.999805 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:13.000262 (+ 457us) server_negotiation.cc:299] Negotiation successful
0928 14:57:13.000511 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":946,"thread_start_us":790,"threads_started":1}
I20260928 14:57:13.001034 7649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:12.978623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:32977 (local address 127.0.0.1:42446)
0928 14:57:12.979455 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:12.979483 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:12.979684 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:12.980279 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:12.980284 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:12.981248 (+ 964us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:12.981615 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:12.981630 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:12.983559 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:12.983567 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:12.985178 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:12.985198 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:12.986058 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:12.986066 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:12.986259 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:12.992002 (+ 5743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:12.992034 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:12.994956 (+ 2922us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:12.998243 (+ 3287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:12.998260 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:12.998269 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:12.998736 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:12.999111 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:12.999116 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:12.999120 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:12.999259 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:12.999784 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:12.999792 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:13.000011 (+ 219us) client_negotiation.cc:769] Sending connection context
0928 14:57:13.000175 (+ 164us) client_negotiation.cc:240] Negotiation successful
0928 14:57:13.000632 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"mutex_wait_us":165,"thread_start_us":403,"threads_started":1}
I20260928 14:57:13.006860 7102 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:42446:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 14:57:13.007977 7102 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:57:13.015408 7239 tablet_service.cc:1467] Processing CreateTablet for tablet 1ec7a1258ca6451d824026fe65c72b66 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1e2b70aca73e4aa18ee711bd68a05086]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:57:13.015795 7379 tablet_service.cc:1467] Processing CreateTablet for tablet 1ec7a1258ca6451d824026fe65c72b66 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1e2b70aca73e4aa18ee711bd68a05086]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:57:13.016157 7239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ec7a1258ca6451d824026fe65c72b66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:13.016403 7379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ec7a1258ca6451d824026fe65c72b66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:13.020680 7517 tablet_service.cc:1467] Processing CreateTablet for tablet 1ec7a1258ca6451d824026fe65c72b66 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1e2b70aca73e4aa18ee711bd68a05086]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:57:13.021222 7517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ec7a1258ca6451d824026fe65c72b66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:13.022437 7612 tablet_bootstrap.cc:492] T 1ec7a1258ca6451d824026fe65c72b66 P 9659e13c2242471c8899cf4f87d3fa36: Bootstrap starting.
I20260928 14:57:13.023699 7613 tablet_bootstrap.cc:492] T 1ec7a1258ca6451d824026fe65c72b66 P 48c7f448d8ae483ca98628d00e3b6fcc: Bootstrap starting.
I20260928 14:57:13.024040 7612 tablet_bootstrap.cc:654] T 1ec7a1258ca6451d824026fe65c72b66 P 9659e13c2242471c8899cf4f87d3fa36: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:13.025498 7613 tablet_bootstrap.cc:654] T 1ec7a1258ca6451d824026fe65c72b66 P 48c7f448d8ae483ca98628d00e3b6fcc: Neither blocks nor log segments found. Creating new log.
W20260928 14:57:13.026489 7310 tablet.cc:2369] T 61518ef22470421f98bcfe471aee53f3 P 9659e13c2242471c8899cf4f87d3fa36: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:57:13.026964 7612 tablet_bootstrap.cc:492] T 1ec7a1258ca6451d824026fe65c72b66 P 9659e13c2242471c8899cf4f87d3fa36: No bootstrap required, opened a new log
I20260928 14:57:13.027158 7612 ts_tablet_manager.cc:1397] T 1ec7a1258ca6451d824026fe65c72b66 P 9659e13c2242471c8899cf4f87d3fa36: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20260928 14:57:13.027910 7614 tablet_bootstrap.cc:492] T 1ec7a1258ca6451d824026fe65c72b66 P 2a617e1e19a24cbc8c99e83995afc049: Bootstrap starting.
I20260928 14:57:13.029004 7612 raft_consensus.cc:348] T 1ec7a1258ca6451d824026fe65c72b66 P 9659e13c2242471c8899cf4f87d3fa36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 } } peers { permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 } } peers { permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 } }
I20260928 14:57:13.029546 7612 raft_consensus.cc:374] T 1ec7a1258ca6451d824026fe65c72b66 P 9659e13c2242471c8899cf4f87d3fa36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:13.029592 7614 tablet_bootstrap.cc:654] T 1ec7a1258ca6451d824026fe65c72b66 P 2a617e1e19a24cbc8c99e83995afc049: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:13.029732 7612 raft_consensus.cc:729] T 1ec7a1258ca6451d824026fe65c72b66 P 9659e13c2242471c8899cf4f87d3fa36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9659e13c2242471c8899cf4f87d3fa36, State: Initialized, Role: FOLLOWER
I20260928 14:57:13.030144 7612 consensus_queue.cc:260] T 1ec7a1258ca6451d824026fe65c72b66 P 9659e13c2242471c8899cf4f87d3fa36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 } } peers { permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 } } peers { permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 } }
I20260928 14:57:13.030826 7618 tablet_replica.cc:439] T 1ec7a1258ca6451d824026fe65c72b66 P 9659e13c2242471c8899cf4f87d3fa36: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 } } peers { permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 } } peers { permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 } } }
I20260928 14:57:13.031207 7618 tablet_replica.cc:442] T 1ec7a1258ca6451d824026fe65c72b66 P 9659e13c2242471c8899cf4f87d3fa36: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:57:13.035759 7612 ts_tablet_manager.cc:1428] T 1ec7a1258ca6451d824026fe65c72b66 P 9659e13c2242471c8899cf4f87d3fa36: Time spent starting tablet: real 0.008s user 0.001s sys 0.002s
I20260928 14:57:13.040040 7613 tablet_bootstrap.cc:492] T 1ec7a1258ca6451d824026fe65c72b66 P 48c7f448d8ae483ca98628d00e3b6fcc: No bootstrap required, opened a new log
I20260928 14:57:13.040256 7613 ts_tablet_manager.cc:1397] T 1ec7a1258ca6451d824026fe65c72b66 P 48c7f448d8ae483ca98628d00e3b6fcc: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.008s
I20260928 14:57:13.040898 7613 raft_consensus.cc:348] T 1ec7a1258ca6451d824026fe65c72b66 P 48c7f448d8ae483ca98628d00e3b6fcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 } } peers { permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 } } peers { permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 } }
I20260928 14:57:13.041154 7613 raft_consensus.cc:374] T 1ec7a1258ca6451d824026fe65c72b66 P 48c7f448d8ae483ca98628d00e3b6fcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:13.041246 7613 raft_consensus.cc:729] T 1ec7a1258ca6451d824026fe65c72b66 P 48c7f448d8ae483ca98628d00e3b6fcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48c7f448d8ae483ca98628d00e3b6fcc, State: Initialized, Role: FOLLOWER
I20260928 14:57:13.041523 7613 consensus_queue.cc:260] T 1ec7a1258ca6451d824026fe65c72b66 P 48c7f448d8ae483ca98628d00e3b6fcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 } } peers { permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 } } peers { permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 } }
I20260928 14:57:13.042177 7613 ts_tablet_manager.cc:1428] T 1ec7a1258ca6451d824026fe65c72b66 P 48c7f448d8ae483ca98628d00e3b6fcc: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260928 14:57:13.042012 7619 tablet_replica.cc:439] T 1ec7a1258ca6451d824026fe65c72b66 P 48c7f448d8ae483ca98628d00e3b6fcc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 } } peers { permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 } } peers { permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 } } }
I20260928 14:57:13.042362 7619 tablet_replica.cc:442] T 1ec7a1258ca6451d824026fe65c72b66 P 48c7f448d8ae483ca98628d00e3b6fcc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:57:13.043061 7614 tablet_bootstrap.cc:492] T 1ec7a1258ca6451d824026fe65c72b66 P 2a617e1e19a24cbc8c99e83995afc049: No bootstrap required, opened a new log
I20260928 14:57:13.043296 7614 ts_tablet_manager.cc:1397] T 1ec7a1258ca6451d824026fe65c72b66 P 2a617e1e19a24cbc8c99e83995afc049: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.003s
I20260928 14:57:13.044833 7614 raft_consensus.cc:348] T 1ec7a1258ca6451d824026fe65c72b66 P 2a617e1e19a24cbc8c99e83995afc049 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 } } peers { permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 } } peers { permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 } }
I20260928 14:57:13.045218 7614 raft_consensus.cc:374] T 1ec7a1258ca6451d824026fe65c72b66 P 2a617e1e19a24cbc8c99e83995afc049 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:13.045343 7614 raft_consensus.cc:729] T 1ec7a1258ca6451d824026fe65c72b66 P 2a617e1e19a24cbc8c99e83995afc049 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a617e1e19a24cbc8c99e83995afc049, State: Initialized, Role: FOLLOWER
I20260928 14:57:13.045655 7614 consensus_queue.cc:260] T 1ec7a1258ca6451d824026fe65c72b66 P 2a617e1e19a24cbc8c99e83995afc049 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 } } peers { permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 } } peers { permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 } }
I20260928 14:57:13.046222 7620 tablet_replica.cc:439] T 1ec7a1258ca6451d824026fe65c72b66 P 2a617e1e19a24cbc8c99e83995afc049: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 } } peers { permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 } } peers { permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 } } }
I20260928 14:57:13.046612 7620 tablet_replica.cc:442] T 1ec7a1258ca6451d824026fe65c72b66 P 2a617e1e19a24cbc8c99e83995afc049: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:57:13.047297 7614 ts_tablet_manager.cc:1428] T 1ec7a1258ca6451d824026fe65c72b66 P 2a617e1e19a24cbc8c99e83995afc049: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260928 14:57:13.270706 7620 raft_consensus.cc:482] T 1ec7a1258ca6451d824026fe65c72b66 P 2a617e1e19a24cbc8c99e83995afc049 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:57:13.270916 7620 raft_consensus.cc:504] T 1ec7a1258ca6451d824026fe65c72b66 P 2a617e1e19a24cbc8c99e83995afc049 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 } } peers { permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 } } peers { permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 } }
I20260928 14:57:13.272729 7620 leader_election.cc:288] T 1ec7a1258ca6451d824026fe65c72b66 P 2a617e1e19a24cbc8c99e83995afc049 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9659e13c2242471c8899cf4f87d3fa36 (127.31.40.1:43671), 48c7f448d8ae483ca98628d00e3b6fcc (127.31.40.2:34247)
I20260928 14:57:13.290189 7656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:13.273220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:52937 (local address 127.31.40.2:34247)
0928 14:57:13.284381 (+ 11161us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:13.284388 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:13.284419 (+ 31us) server_negotiation.cc:407] Connection header received
0928 14:57:13.284594 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:13.284597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:13.284658 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:13.284784 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:13.285338 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.286433 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.288430 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.289205 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.289364 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:13.289474 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:13.289746 (+ 272us) server_negotiation.cc:299] Negotiation successful
0928 14:57:13.289885 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11011,"thread_start_us":10266,"threads_started":1}
I20260928 14:57:13.290232 7657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:13.272932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.2:34247 (local address 127.31.40.3:52937)
0928 14:57:13.284292 (+ 11360us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:13.284312 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:13.284441 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:13.284806 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:13.284811 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:13.284835 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:13.285093 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:13.285111 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.286605 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.286610 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:13.288240 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:13.288253 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.289396 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.289403 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:13.289581 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:13.289605 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 14:57:13.289718 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 14:57:13.289865 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11217,"mutex_wait_us":461,"thread_start_us":10605,"threads_started":1}
I20260928 14:57:13.291189 7399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ec7a1258ca6451d824026fe65c72b66" candidate_uuid: "2a617e1e19a24cbc8c99e83995afc049" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" is_pre_election: true
I20260928 14:57:13.291466 7399 raft_consensus.cc:2454] T 1ec7a1258ca6451d824026fe65c72b66 P 48c7f448d8ae483ca98628d00e3b6fcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a617e1e19a24cbc8c99e83995afc049 in term 0.
I20260928 14:57:13.291682 7655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:13.273178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:40215 (local address 127.31.40.1:43671)
0928 14:57:13.285372 (+ 12194us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:13.285377 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:13.285393 (+ 16us) server_negotiation.cc:407] Connection header received
0928 14:57:13.285448 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:13.285452 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:13.285509 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:13.285625 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:13.286077 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.286941 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.289383 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.290711 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.290832 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:13.290931 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:13.291244 (+ 313us) server_negotiation.cc:299] Negotiation successful
0928 14:57:13.291387 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12075,"thread_start_us":11968,"threads_started":1}
I20260928 14:57:13.291682 7654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:13.272932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:43671 (local address 127.31.40.3:40215)
0928 14:57:13.284040 (+ 11108us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:13.284067 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:13.284250 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:13.285645 (+ 1395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:13.285649 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:13.285672 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:13.285915 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:13.285923 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.287111 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.287117 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:13.289179 (+ 2062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:13.289194 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.290883 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.290889 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:13.291075 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:13.291101 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 14:57:13.291214 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 14:57:13.291357 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10978,"thread_start_us":10826,"threads_started":1}
I20260928 14:57:13.292222 7470 leader_election.cc:302] T 1ec7a1258ca6451d824026fe65c72b66 P 2a617e1e19a24cbc8c99e83995afc049 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a617e1e19a24cbc8c99e83995afc049, 48c7f448d8ae483ca98628d00e3b6fcc; no voters:
I20260928 14:57:13.292554 7259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ec7a1258ca6451d824026fe65c72b66" candidate_uuid: "2a617e1e19a24cbc8c99e83995afc049" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9659e13c2242471c8899cf4f87d3fa36" is_pre_election: true
I20260928 14:57:13.292680 7620 raft_consensus.cc:2781] T 1ec7a1258ca6451d824026fe65c72b66 P 2a617e1e19a24cbc8c99e83995afc049 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:57:13.292794 7620 raft_consensus.cc:482] T 1ec7a1258ca6451d824026fe65c72b66 P 2a617e1e19a24cbc8c99e83995afc049 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:57:13.292881 7620 raft_consensus.cc:3037] T 1ec7a1258ca6451d824026fe65c72b66 P 2a617e1e19a24cbc8c99e83995afc049 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:13.292858 7259 raft_consensus.cc:2454] T 1ec7a1258ca6451d824026fe65c72b66 P 9659e13c2242471c8899cf4f87d3fa36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a617e1e19a24cbc8c99e83995afc049 in term 0.
I20260928 14:57:13.295935 7620 raft_consensus.cc:504] T 1ec7a1258ca6451d824026fe65c72b66 P 2a617e1e19a24cbc8c99e83995afc049 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 } } peers { permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 } } peers { permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 } }
I20260928 14:57:13.296602 7620 leader_election.cc:288] T 1ec7a1258ca6451d824026fe65c72b66 P 2a617e1e19a24cbc8c99e83995afc049 [CANDIDATE]: Term 1 election: Requested vote from peers 9659e13c2242471c8899cf4f87d3fa36 (127.31.40.1:43671), 48c7f448d8ae483ca98628d00e3b6fcc (127.31.40.2:34247)
I20260928 14:57:13.297179 7259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ec7a1258ca6451d824026fe65c72b66" candidate_uuid: "2a617e1e19a24cbc8c99e83995afc049" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9659e13c2242471c8899cf4f87d3fa36"
I20260928 14:57:13.297241 7399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ec7a1258ca6451d824026fe65c72b66" candidate_uuid: "2a617e1e19a24cbc8c99e83995afc049" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48c7f448d8ae483ca98628d00e3b6fcc"
I20260928 14:57:13.297399 7259 raft_consensus.cc:3037] T 1ec7a1258ca6451d824026fe65c72b66 P 9659e13c2242471c8899cf4f87d3fa36 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:13.297542 7399 raft_consensus.cc:3037] T 1ec7a1258ca6451d824026fe65c72b66 P 48c7f448d8ae483ca98628d00e3b6fcc [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:13.299504 7399 raft_consensus.cc:2454] T 1ec7a1258ca6451d824026fe65c72b66 P 48c7f448d8ae483ca98628d00e3b6fcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a617e1e19a24cbc8c99e83995afc049 in term 1.
I20260928 14:57:13.300165 7470 leader_election.cc:302] T 1ec7a1258ca6451d824026fe65c72b66 P 2a617e1e19a24cbc8c99e83995afc049 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a617e1e19a24cbc8c99e83995afc049, 48c7f448d8ae483ca98628d00e3b6fcc; no voters:
I20260928 14:57:13.300498 7620 raft_consensus.cc:2781] T 1ec7a1258ca6451d824026fe65c72b66 P 2a617e1e19a24cbc8c99e83995afc049 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:57:13.300794 7259 raft_consensus.cc:2454] T 1ec7a1258ca6451d824026fe65c72b66 P 9659e13c2242471c8899cf4f87d3fa36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a617e1e19a24cbc8c99e83995afc049 in term 1.
I20260928 14:57:13.301208 7620 raft_consensus.cc:686] T 1ec7a1258ca6451d824026fe65c72b66 P 2a617e1e19a24cbc8c99e83995afc049 [term 1 LEADER]: Becoming Leader. State: Replica: 2a617e1e19a24cbc8c99e83995afc049, State: Running, Role: LEADER
I20260928 14:57:13.301649 7620 consensus_queue.cc:237] T 1ec7a1258ca6451d824026fe65c72b66 P 2a617e1e19a24cbc8c99e83995afc049 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 } } peers { permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 } } peers { permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 } }
I20260928 14:57:13.304111 7658 tablet_replica.cc:439] T 1ec7a1258ca6451d824026fe65c72b66 P 2a617e1e19a24cbc8c99e83995afc049: TxnStatusTablet state changed. Reason: New leader 2a617e1e19a24cbc8c99e83995afc049. Latest consensus state: current_term: 1 leader_uuid: "2a617e1e19a24cbc8c99e83995afc049" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 } } peers { permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 } } peers { permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 } } }
I20260928 14:57:13.304497 7658 tablet_replica.cc:442] T 1ec7a1258ca6451d824026fe65c72b66 P 2a617e1e19a24cbc8c99e83995afc049: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:57:13.305146 7660 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 14:57:13.305375 7660 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 14:57:13.305795 7101 catalog_manager.cc:5495] T 1ec7a1258ca6451d824026fe65c72b66 P 2a617e1e19a24cbc8c99e83995afc049 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a617e1e19a24cbc8c99e83995afc049 (127.31.40.3). New cstate: current_term: 1 leader_uuid: "2a617e1e19a24cbc8c99e83995afc049" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 } health_report { overall_health: HEALTHY } } }
I20260928 14:57:13.333689 7663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:13.326923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55138 (local address 127.31.40.3:35371)
0928 14:57:13.327546 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:13.327550 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:13.327572 (+ 22us) server_negotiation.cc:407] Connection header received
0928 14:57:13.327667 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:13.327671 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:13.327759 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:13.327989 (+ 230us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:57:13.328498 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.329367 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.330737 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.331247 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.331437 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:13.332219 (+ 782us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:57:13.332921 (+ 702us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:57:13.333055 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:13.333281 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 14:57:13.333370 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":331,"threads_started":1}
I20260928 14:57:13.333701 7649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:13.326789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.3:35371 (local address 127.0.0.1:55138)
0928 14:57:13.327053 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:13.327076 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:13.327246 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:13.328029 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:13.328034 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:13.328049 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:57:13.328283 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:13.328293 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.329534 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.329540 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:13.330572 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:13.330585 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.331536 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.331544 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:13.331788 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:13.331908 (+ 120us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:57:13.333057 (+ 1149us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:57:13.333088 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 14:57:13.333196 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 14:57:13.333385 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 14:57:13.338855 7399 raft_consensus.cc:1261] T 1ec7a1258ca6451d824026fe65c72b66 P 48c7f448d8ae483ca98628d00e3b6fcc [term 1 FOLLOWER]: Refusing update from remote peer 2a617e1e19a24cbc8c99e83995afc049: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:57:13.339033 7259 raft_consensus.cc:1261] T 1ec7a1258ca6451d824026fe65c72b66 P 9659e13c2242471c8899cf4f87d3fa36 [term 1 FOLLOWER]: Refusing update from remote peer 2a617e1e19a24cbc8c99e83995afc049: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:57:13.339684 7620 consensus_queue.cc:1035] T 1ec7a1258ca6451d824026fe65c72b66 P 2a617e1e19a24cbc8c99e83995afc049 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:57:13.340075 7658 consensus_queue.cc:1035] T 1ec7a1258ca6451d824026fe65c72b66 P 2a617e1e19a24cbc8c99e83995afc049 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:57:13.349988 7625 consensus_queue.cc:1035] T 61518ef22470421f98bcfe471aee53f3 P 9659e13c2242471c8899cf4f87d3fa36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:57:13.364878 7619 tablet_replica.cc:439] T 1ec7a1258ca6451d824026fe65c72b66 P 48c7f448d8ae483ca98628d00e3b6fcc: TxnStatusTablet state changed. Reason: New leader 2a617e1e19a24cbc8c99e83995afc049. Latest consensus state: current_term: 1 leader_uuid: "2a617e1e19a24cbc8c99e83995afc049" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 } } peers { permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 } } peers { permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 } } }
I20260928 14:57:13.365233 7619 tablet_replica.cc:442] T 1ec7a1258ca6451d824026fe65c72b66 P 48c7f448d8ae483ca98628d00e3b6fcc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:57:13.369994 7665 mvcc.cc:205] Tried to move back new op lower bound from 7334328046947782656 to 7334328046807871488. Current Snapshot: MvccSnapshot[applied={T|T < 7334328046947782656}]
I20260928 14:57:13.370445 7619 tablet_replica.cc:439] T 1ec7a1258ca6451d824026fe65c72b66 P 48c7f448d8ae483ca98628d00e3b6fcc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2a617e1e19a24cbc8c99e83995afc049" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 } } peers { permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 } } peers { permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 } } }
I20260928 14:57:13.370749 7619 tablet_replica.cc:442] T 1ec7a1258ca6451d824026fe65c72b66 P 48c7f448d8ae483ca98628d00e3b6fcc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:57:13.385520 7620 tablet_replica.cc:439] T 1ec7a1258ca6451d824026fe65c72b66 P 2a617e1e19a24cbc8c99e83995afc049: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2a617e1e19a24cbc8c99e83995afc049" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 } } peers { permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 } } peers { permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 } } }
I20260928 14:57:13.385828 7620 tablet_replica.cc:442] T 1ec7a1258ca6451d824026fe65c72b66 P 2a617e1e19a24cbc8c99e83995afc049: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:57:13.401839 7673 consensus_queue.cc:1035] T 61518ef22470421f98bcfe471aee53f3 P 9659e13c2242471c8899cf4f87d3fa36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:57:13.408635 7620 tablet_replica.cc:439] T 1ec7a1258ca6451d824026fe65c72b66 P 2a617e1e19a24cbc8c99e83995afc049: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2a617e1e19a24cbc8c99e83995afc049" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 } } peers { permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 } } peers { permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 } } }
I20260928 14:57:13.408939 7620 tablet_replica.cc:442] T 1ec7a1258ca6451d824026fe65c72b66 P 2a617e1e19a24cbc8c99e83995afc049: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:57:13.406706 7618 tablet_replica.cc:439] T 1ec7a1258ca6451d824026fe65c72b66 P 9659e13c2242471c8899cf4f87d3fa36: TxnStatusTablet state changed. Reason: New leader 2a617e1e19a24cbc8c99e83995afc049. Latest consensus state: current_term: 1 leader_uuid: "2a617e1e19a24cbc8c99e83995afc049" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 } } peers { permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 } } peers { permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 } } }
I20260928 14:57:13.416098 7618 tablet_replica.cc:442] T 1ec7a1258ca6451d824026fe65c72b66 P 9659e13c2242471c8899cf4f87d3fa36: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:57:13.424355 7675 tablet_replica.cc:439] T 1ec7a1258ca6451d824026fe65c72b66 P 9659e13c2242471c8899cf4f87d3fa36: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2a617e1e19a24cbc8c99e83995afc049" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 } } peers { permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 } } peers { permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 } } }
I20260928 14:57:13.425112 7675 tablet_replica.cc:442] T 1ec7a1258ca6451d824026fe65c72b66 P 9659e13c2242471c8899cf4f87d3fa36: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:57:13.446724 7663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:13.433114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:47279 (local address 127.31.40.3:35371)
0928 14:57:13.433307 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:13.433313 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:13.439577 (+ 6264us) server_negotiation.cc:407] Connection header received
0928 14:57:13.439779 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:13.439785 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:13.439918 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:13.440028 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:57:13.440636 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.442013 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.444185 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.444966 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.445155 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:13.445717 (+ 562us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:57:13.445889 (+ 172us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:57:13.445988 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:13.446216 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 14:57:13.446320 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 14:57:13.447625 7680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:13.432504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.3:35371 (local address 127.31.40.1:47279)
0928 14:57:13.439433 (+ 6929us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:13.439456 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:13.439630 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:13.440063 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:13.440070 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:13.440086 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:57:13.440444 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:13.440456 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.442195 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.442202 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:13.444025 (+ 1823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:13.444041 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.445194 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.445201 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:13.445392 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:13.445516 (+ 124us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:57:13.446049 (+ 533us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:57:13.446062 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 14:57:13.447036 (+ 974us) client_negotiation.cc:240] Negotiation successful
0928 14:57:13.447213 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6710,"thread_start_us":6552,"threads_started":1}
I20260928 14:57:13.499277 7655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:13.492496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:57149 (local address 127.31.40.1:43671)
0928 14:57:13.492703 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:13.492711 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:13.493527 (+ 816us) server_negotiation.cc:407] Connection header received
0928 14:57:13.493762 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:13.493767 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:13.493870 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:13.493972 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:57:13.494540 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.495278 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.497275 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.497938 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.498067 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:13.498504 (+ 437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:57:13.498641 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:57:13.498738 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:13.498914 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 14:57:13.499001 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 14:57:13.499303 7683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:13.492531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:43671 (local address 127.31.40.3:57149)
0928 14:57:13.493410 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:13.493426 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:13.493614 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:13.493987 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:13.493992 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:13.494003 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:57:13.494384 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:13.494394 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.495414 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.495420 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:13.497116 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:13.497130 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.498118 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.498123 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:13.498254 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:13.498313 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:57:13.498769 (+ 456us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:57:13.498781 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 14:57:13.498860 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 14:57:13.498972 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"thread_start_us":526,"threads_started":1}
I20260928 14:57:13.537954 7102 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:57:13.538255 7102 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:57:13.541551 7102 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 73109dfa9b4d4463a096a15f0501a28c: Sending DeleteTablet for 3 replicas of tablet 61518ef22470421f98bcfe471aee53f3
I20260928 14:57:13.542573 7379 tablet_service.cc:1514] Processing DeleteTablet for tablet 61518ef22470421f98bcfe471aee53f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:57:13 UTC) from {username='oryx', principal='oryx/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:35654
I20260928 14:57:13.544018 7517 tablet_service.cc:1514] Processing DeleteTablet for tablet 61518ef22470421f98bcfe471aee53f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:57:13 UTC) from {username='oryx', principal='oryx/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:55132
I20260928 14:57:13.548228 7239 tablet_service.cc:1514] Processing DeleteTablet for tablet 61518ef22470421f98bcfe471aee53f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:57:13 UTC) from {username='oryx', principal='oryx/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:51354
I20260928 14:57:13.555716 7687 tablet_replica.cc:330] T 61518ef22470421f98bcfe471aee53f3 P 2a617e1e19a24cbc8c99e83995afc049: stopping tablet replica
I20260928 14:57:13.555706 7686 tablet_replica.cc:330] T 61518ef22470421f98bcfe471aee53f3 P 48c7f448d8ae483ca98628d00e3b6fcc: stopping tablet replica
I20260928 14:57:13.556043 7687 raft_consensus.cc:2229] T 61518ef22470421f98bcfe471aee53f3 P 2a617e1e19a24cbc8c99e83995afc049 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:57:13.556314 7686 raft_consensus.cc:2229] T 61518ef22470421f98bcfe471aee53f3 P 48c7f448d8ae483ca98628d00e3b6fcc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:57:13.556614 7687 raft_consensus.cc:2258] T 61518ef22470421f98bcfe471aee53f3 P 2a617e1e19a24cbc8c99e83995afc049 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:57:13.556794 7686 raft_consensus.cc:2258] T 61518ef22470421f98bcfe471aee53f3 P 48c7f448d8ae483ca98628d00e3b6fcc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:57:13.558110 7687 ts_tablet_manager.cc:1905] T 61518ef22470421f98bcfe471aee53f3 P 2a617e1e19a24cbc8c99e83995afc049: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:57:13.558115 7686 ts_tablet_manager.cc:1905] T 61518ef22470421f98bcfe471aee53f3 P 48c7f448d8ae483ca98628d00e3b6fcc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:57:13.566572 7688 tablet_replica.cc:330] T 61518ef22470421f98bcfe471aee53f3 P 9659e13c2242471c8899cf4f87d3fa36: stopping tablet replica
I20260928 14:57:13.566813 7688 raft_consensus.cc:2229] T 61518ef22470421f98bcfe471aee53f3 P 9659e13c2242471c8899cf4f87d3fa36 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:57:13.567269 7688 raft_consensus.cc:2258] T 61518ef22470421f98bcfe471aee53f3 P 9659e13c2242471c8899cf4f87d3fa36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:57:13.567469 7686 ts_tablet_manager.cc:1918] T 61518ef22470421f98bcfe471aee53f3 P 48c7f448d8ae483ca98628d00e3b6fcc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:57:13.567721 7686 log.cc:1196] T 61518ef22470421f98bcfe471aee53f3 P 48c7f448d8ae483ca98628d00e3b6fcc: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-1/wal/wals/61518ef22470421f98bcfe471aee53f3
I20260928 14:57:13.568357 7686 ts_tablet_manager.cc:1939] T 61518ef22470421f98bcfe471aee53f3 P 48c7f448d8ae483ca98628d00e3b6fcc: Deleting consensus metadata
I20260928 14:57:13.568485 7687 ts_tablet_manager.cc:1918] T 61518ef22470421f98bcfe471aee53f3 P 2a617e1e19a24cbc8c99e83995afc049: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:57:13.568660 7687 log.cc:1196] T 61518ef22470421f98bcfe471aee53f3 P 2a617e1e19a24cbc8c99e83995afc049: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-2/wal/wals/61518ef22470421f98bcfe471aee53f3
I20260928 14:57:13.568786 7688 ts_tablet_manager.cc:1905] T 61518ef22470421f98bcfe471aee53f3 P 9659e13c2242471c8899cf4f87d3fa36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:57:13.569204 7687 ts_tablet_manager.cc:1939] T 61518ef22470421f98bcfe471aee53f3 P 2a617e1e19a24cbc8c99e83995afc049: Deleting consensus metadata
I20260928 14:57:13.569947 7087 catalog_manager.cc:4841] TS 48c7f448d8ae483ca98628d00e3b6fcc (127.31.40.2:34247): tablet 61518ef22470421f98bcfe471aee53f3 (table test-table [id=bedf9334a1c24927a09086a6cf7e1129]) successfully deleted
I20260928 14:57:13.570415 7086 catalog_manager.cc:4841] TS 2a617e1e19a24cbc8c99e83995afc049 (127.31.40.3:35371): tablet 61518ef22470421f98bcfe471aee53f3 (table test-table [id=bedf9334a1c24927a09086a6cf7e1129]) successfully deleted
I20260928 14:57:13.576889 7688 ts_tablet_manager.cc:1918] T 61518ef22470421f98bcfe471aee53f3 P 9659e13c2242471c8899cf4f87d3fa36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:57:13.577075 7688 log.cc:1196] T 61518ef22470421f98bcfe471aee53f3 P 9659e13c2242471c8899cf4f87d3fa36: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-0/wal/wals/61518ef22470421f98bcfe471aee53f3
I20260928 14:57:13.577497 7688 ts_tablet_manager.cc:1939] T 61518ef22470421f98bcfe471aee53f3 P 9659e13c2242471c8899cf4f87d3fa36: Deleting consensus metadata
I20260928 14:57:13.578801 7086 catalog_manager.cc:4841] TS 9659e13c2242471c8899cf4f87d3fa36 (127.31.40.1:43671): tablet 61518ef22470421f98bcfe471aee53f3 (table test-table [id=bedf9334a1c24927a09086a6cf7e1129]) successfully deleted
Sep 28 14:57:13 dist-test-slave-j968 krb5kdc[7057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 14:57:13 dist-test-slave-j968 krb5kdc[7057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:57:13 dist-test-slave-j968 krb5kdc[7057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 14:57:13.627969 7695 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:13.608697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42454 (local address 127.31.40.62:32977)
0928 14:57:13.609283 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:13.609293 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:13.609540 (+ 247us) server_negotiation.cc:407] Connection header received
0928 14:57:13.609765 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:13.609769 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:13.609837 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:13.609957 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:13.611308 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.612312 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.614404 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.615110 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.615239 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:13.627622 (+ 12383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42454: BlockingRecv error: recv got EOF from 127.0.0.1:42454 (error 108)
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":247,"threads_started":1}
Sep 28 14:57:13 dist-test-slave-j968 krb5kdc[7057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM
I20260928 14:57:13.654055 7695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:13.635535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42460 (local address 127.31.40.62:32977)
0928 14:57:13.635711 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:13.635718 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:13.636177 (+ 459us) server_negotiation.cc:407] Connection header received
0928 14:57:13.636373 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:13.636378 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:13.636459 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:13.636567 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:13.638483 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.639765 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.642055 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.642501 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.642715 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:13.648204 (+ 5489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:13.648210 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:13.648213 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:13.648254 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:13.651090 (+ 2836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:13.651937 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:13.651942 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:13.651944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:13.652086 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:13.652504 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:13.652512 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:13.652515 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:13.652831 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:13.653121 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:13.653437 (+ 316us) server_negotiation.cc:299] Negotiation successful
0928 14:57:13.653689 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 14:57:13.658715 7102 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42460:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:57:13.659694 7102 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:57:13.667428 7239 tablet_service.cc:1467] Processing CreateTablet for tablet 8b03be264b9d42bca5e06b6da9054922 (DEFAULT_TABLE table=test-table [id=61102a528fa04ab1b4b02846dfe476bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:57:13.667954 7239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b03be264b9d42bca5e06b6da9054922. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:13.667814 7517 tablet_service.cc:1467] Processing CreateTablet for tablet 8b03be264b9d42bca5e06b6da9054922 (DEFAULT_TABLE table=test-table [id=61102a528fa04ab1b4b02846dfe476bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:57:13.668344 7517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b03be264b9d42bca5e06b6da9054922. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:13.668315 7379 tablet_service.cc:1467] Processing CreateTablet for tablet 8b03be264b9d42bca5e06b6da9054922 (DEFAULT_TABLE table=test-table [id=61102a528fa04ab1b4b02846dfe476bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:57:13.668740 7379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b03be264b9d42bca5e06b6da9054922. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:13.679585 7703 tablet_bootstrap.cc:492] T 8b03be264b9d42bca5e06b6da9054922 P 48c7f448d8ae483ca98628d00e3b6fcc: Bootstrap starting.
I20260928 14:57:13.681734 7703 tablet_bootstrap.cc:654] T 8b03be264b9d42bca5e06b6da9054922 P 48c7f448d8ae483ca98628d00e3b6fcc: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:13.683137 7704 tablet_bootstrap.cc:492] T 8b03be264b9d42bca5e06b6da9054922 P 2a617e1e19a24cbc8c99e83995afc049: Bootstrap starting.
I20260928 14:57:13.683535 7702 tablet_bootstrap.cc:492] T 8b03be264b9d42bca5e06b6da9054922 P 9659e13c2242471c8899cf4f87d3fa36: Bootstrap starting.
I20260928 14:57:13.685593 7702 tablet_bootstrap.cc:654] T 8b03be264b9d42bca5e06b6da9054922 P 9659e13c2242471c8899cf4f87d3fa36: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:13.686206 7703 tablet_bootstrap.cc:492] T 8b03be264b9d42bca5e06b6da9054922 P 48c7f448d8ae483ca98628d00e3b6fcc: No bootstrap required, opened a new log
I20260928 14:57:13.686420 7703 ts_tablet_manager.cc:1397] T 8b03be264b9d42bca5e06b6da9054922 P 48c7f448d8ae483ca98628d00e3b6fcc: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20260928 14:57:13.687310 7703 raft_consensus.cc:348] T 8b03be264b9d42bca5e06b6da9054922 P 48c7f448d8ae483ca98628d00e3b6fcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 } } peers { permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 } } peers { permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 } }
I20260928 14:57:13.687579 7703 raft_consensus.cc:374] T 8b03be264b9d42bca5e06b6da9054922 P 48c7f448d8ae483ca98628d00e3b6fcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:13.687680 7703 raft_consensus.cc:729] T 8b03be264b9d42bca5e06b6da9054922 P 48c7f448d8ae483ca98628d00e3b6fcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48c7f448d8ae483ca98628d00e3b6fcc, State: Initialized, Role: FOLLOWER
I20260928 14:57:13.688050 7702 tablet_bootstrap.cc:492] T 8b03be264b9d42bca5e06b6da9054922 P 9659e13c2242471c8899cf4f87d3fa36: No bootstrap required, opened a new log
I20260928 14:57:13.687939 7703 consensus_queue.cc:260] T 8b03be264b9d42bca5e06b6da9054922 P 48c7f448d8ae483ca98628d00e3b6fcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 } } peers { permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 } } peers { permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 } }
I20260928 14:57:13.688524 7703 ts_tablet_manager.cc:1428] T 8b03be264b9d42bca5e06b6da9054922 P 48c7f448d8ae483ca98628d00e3b6fcc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:57:13.689111 7702 ts_tablet_manager.cc:1397] T 8b03be264b9d42bca5e06b6da9054922 P 9659e13c2242471c8899cf4f87d3fa36: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 14:57:13.690032 7702 raft_consensus.cc:348] T 8b03be264b9d42bca5e06b6da9054922 P 9659e13c2242471c8899cf4f87d3fa36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 } } peers { permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 } } peers { permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 } }
I20260928 14:57:13.690320 7702 raft_consensus.cc:374] T 8b03be264b9d42bca5e06b6da9054922 P 9659e13c2242471c8899cf4f87d3fa36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:13.690428 7702 raft_consensus.cc:729] T 8b03be264b9d42bca5e06b6da9054922 P 9659e13c2242471c8899cf4f87d3fa36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9659e13c2242471c8899cf4f87d3fa36, State: Initialized, Role: FOLLOWER
I20260928 14:57:13.690721 7702 consensus_queue.cc:260] T 8b03be264b9d42bca5e06b6da9054922 P 9659e13c2242471c8899cf4f87d3fa36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 } } peers { permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 } } peers { permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 } }
I20260928 14:57:13.691376 7702 ts_tablet_manager.cc:1428] T 8b03be264b9d42bca5e06b6da9054922 P 9659e13c2242471c8899cf4f87d3fa36: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260928 14:57:13.692747 7704 tablet_bootstrap.cc:654] T 8b03be264b9d42bca5e06b6da9054922 P 2a617e1e19a24cbc8c99e83995afc049: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:13.695061 7704 tablet_bootstrap.cc:492] T 8b03be264b9d42bca5e06b6da9054922 P 2a617e1e19a24cbc8c99e83995afc049: No bootstrap required, opened a new log
I20260928 14:57:13.695241 7704 ts_tablet_manager.cc:1397] T 8b03be264b9d42bca5e06b6da9054922 P 2a617e1e19a24cbc8c99e83995afc049: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20260928 14:57:13.696143 7704 raft_consensus.cc:348] T 8b03be264b9d42bca5e06b6da9054922 P 2a617e1e19a24cbc8c99e83995afc049 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 } } peers { permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 } } peers { permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 } }
I20260928 14:57:13.696401 7704 raft_consensus.cc:374] T 8b03be264b9d42bca5e06b6da9054922 P 2a617e1e19a24cbc8c99e83995afc049 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:13.696508 7704 raft_consensus.cc:729] T 8b03be264b9d42bca5e06b6da9054922 P 2a617e1e19a24cbc8c99e83995afc049 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a617e1e19a24cbc8c99e83995afc049, State: Initialized, Role: FOLLOWER
I20260928 14:57:13.696802 7704 consensus_queue.cc:260] T 8b03be264b9d42bca5e06b6da9054922 P 2a617e1e19a24cbc8c99e83995afc049 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 } } peers { permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 } } peers { permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 } }
I20260928 14:57:13.697440 7704 ts_tablet_manager.cc:1428] T 8b03be264b9d42bca5e06b6da9054922 P 2a617e1e19a24cbc8c99e83995afc049: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:57:13.783288 7619 raft_consensus.cc:482] T 8b03be264b9d42bca5e06b6da9054922 P 48c7f448d8ae483ca98628d00e3b6fcc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:57:13.783533 7619 raft_consensus.cc:504] T 8b03be264b9d42bca5e06b6da9054922 P 48c7f448d8ae483ca98628d00e3b6fcc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 } } peers { permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 } } peers { permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 } }
I20260928 14:57:13.784360 7619 leader_election.cc:288] T 8b03be264b9d42bca5e06b6da9054922 P 48c7f448d8ae483ca98628d00e3b6fcc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9659e13c2242471c8899cf4f87d3fa36 (127.31.40.1:43671), 2a617e1e19a24cbc8c99e83995afc049 (127.31.40.3:35371)
I20260928 14:57:13.785331 7538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b03be264b9d42bca5e06b6da9054922" candidate_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a617e1e19a24cbc8c99e83995afc049" is_pre_election: true
I20260928 14:57:13.785635 7538 raft_consensus.cc:2454] T 8b03be264b9d42bca5e06b6da9054922 P 2a617e1e19a24cbc8c99e83995afc049 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48c7f448d8ae483ca98628d00e3b6fcc in term 0.
I20260928 14:57:13.786181 7329 leader_election.cc:302] T 8b03be264b9d42bca5e06b6da9054922 P 48c7f448d8ae483ca98628d00e3b6fcc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a617e1e19a24cbc8c99e83995afc049, 48c7f448d8ae483ca98628d00e3b6fcc; no voters:
I20260928 14:57:13.786438 7258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b03be264b9d42bca5e06b6da9054922" candidate_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9659e13c2242471c8899cf4f87d3fa36" is_pre_election: true
I20260928 14:57:13.786607 7619 raft_consensus.cc:2781] T 8b03be264b9d42bca5e06b6da9054922 P 48c7f448d8ae483ca98628d00e3b6fcc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:57:13.786736 7619 raft_consensus.cc:482] T 8b03be264b9d42bca5e06b6da9054922 P 48c7f448d8ae483ca98628d00e3b6fcc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:57:13.786780 7258 raft_consensus.cc:2454] T 8b03be264b9d42bca5e06b6da9054922 P 9659e13c2242471c8899cf4f87d3fa36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48c7f448d8ae483ca98628d00e3b6fcc in term 0.
I20260928 14:57:13.786829 7619 raft_consensus.cc:3037] T 8b03be264b9d42bca5e06b6da9054922 P 48c7f448d8ae483ca98628d00e3b6fcc [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:13.790489 7619 raft_consensus.cc:504] T 8b03be264b9d42bca5e06b6da9054922 P 48c7f448d8ae483ca98628d00e3b6fcc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 } } peers { permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 } } peers { permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 } }
I20260928 14:57:13.791299 7619 leader_election.cc:288] T 8b03be264b9d42bca5e06b6da9054922 P 48c7f448d8ae483ca98628d00e3b6fcc [CANDIDATE]: Term 1 election: Requested vote from peers 9659e13c2242471c8899cf4f87d3fa36 (127.31.40.1:43671), 2a617e1e19a24cbc8c99e83995afc049 (127.31.40.3:35371)
I20260928 14:57:13.792155 7258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b03be264b9d42bca5e06b6da9054922" candidate_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9659e13c2242471c8899cf4f87d3fa36"
I20260928 14:57:13.792404 7258 raft_consensus.cc:3037] T 8b03be264b9d42bca5e06b6da9054922 P 9659e13c2242471c8899cf4f87d3fa36 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:13.792553 7538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b03be264b9d42bca5e06b6da9054922" candidate_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a617e1e19a24cbc8c99e83995afc049"
I20260928 14:57:13.792807 7538 raft_consensus.cc:3037] T 8b03be264b9d42bca5e06b6da9054922 P 2a617e1e19a24cbc8c99e83995afc049 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:13.795671 7258 raft_consensus.cc:2454] T 8b03be264b9d42bca5e06b6da9054922 P 9659e13c2242471c8899cf4f87d3fa36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48c7f448d8ae483ca98628d00e3b6fcc in term 1.
I20260928 14:57:13.796293 7538 raft_consensus.cc:2454] T 8b03be264b9d42bca5e06b6da9054922 P 2a617e1e19a24cbc8c99e83995afc049 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48c7f448d8ae483ca98628d00e3b6fcc in term 1.
I20260928 14:57:13.796303 7329 leader_election.cc:302] T 8b03be264b9d42bca5e06b6da9054922 P 48c7f448d8ae483ca98628d00e3b6fcc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48c7f448d8ae483ca98628d00e3b6fcc, 9659e13c2242471c8899cf4f87d3fa36; no voters:
I20260928 14:57:13.796909 7619 raft_consensus.cc:2781] T 8b03be264b9d42bca5e06b6da9054922 P 48c7f448d8ae483ca98628d00e3b6fcc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:57:13.797447 7619 raft_consensus.cc:686] T 8b03be264b9d42bca5e06b6da9054922 P 48c7f448d8ae483ca98628d00e3b6fcc [term 1 LEADER]: Becoming Leader. State: Replica: 48c7f448d8ae483ca98628d00e3b6fcc, State: Running, Role: LEADER
I20260928 14:57:13.797889 7619 consensus_queue.cc:237] T 8b03be264b9d42bca5e06b6da9054922 P 48c7f448d8ae483ca98628d00e3b6fcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 } } peers { permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 } } peers { permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 } }
I20260928 14:57:13.801740 7102 catalog_manager.cc:5495] T 8b03be264b9d42bca5e06b6da9054922 P 48c7f448d8ae483ca98628d00e3b6fcc reported cstate change: term changed from 0 to 1, leader changed from <none> to 48c7f448d8ae483ca98628d00e3b6fcc (127.31.40.2). New cstate: current_term: 1 leader_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 } health_report { overall_health: UNKNOWN } } }
I20260928 14:57:13.822286 7711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:13.815456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35656 (local address 127.31.40.2:34247)
0928 14:57:13.816116 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:13.816122 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:13.816149 (+ 27us) server_negotiation.cc:407] Connection header received
0928 14:57:13.816268 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:13.816274 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:13.816395 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:13.816602 (+ 207us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:57:13.816978 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.818127 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.819472 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.820181 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.820314 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:13.820729 (+ 415us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:57:13.821518 (+ 789us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:57:13.821753 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:13.821869 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 14:57:13.821965 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":343,"threads_started":1}
I20260928 14:57:13.826475 7258 raft_consensus.cc:1261] T 8b03be264b9d42bca5e06b6da9054922 P 9659e13c2242471c8899cf4f87d3fa36 [term 1 FOLLOWER]: Refusing update from remote peer 48c7f448d8ae483ca98628d00e3b6fcc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:57:13.826582 7538 raft_consensus.cc:1261] T 8b03be264b9d42bca5e06b6da9054922 P 2a617e1e19a24cbc8c99e83995afc049 [term 1 FOLLOWER]: Refusing update from remote peer 48c7f448d8ae483ca98628d00e3b6fcc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:57:13.827306 7619 consensus_queue.cc:1035] T 8b03be264b9d42bca5e06b6da9054922 P 48c7f448d8ae483ca98628d00e3b6fcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:57:13.827690 7708 consensus_queue.cc:1035] T 8b03be264b9d42bca5e06b6da9054922 P 48c7f448d8ae483ca98628d00e3b6fcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:57:13.856122 7102 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:57:13.856423 7102 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:57:13.860123 7102 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 73109dfa9b4d4463a096a15f0501a28c: Sending DeleteTablet for 3 replicas of tablet 8b03be264b9d42bca5e06b6da9054922
I20260928 14:57:13.861330 7239 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b03be264b9d42bca5e06b6da9054922 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:57:13 UTC) from {username='oryx', principal='oryx/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:51354
I20260928 14:57:13.861655 7517 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b03be264b9d42bca5e06b6da9054922 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:57:13 UTC) from {username='oryx', principal='oryx/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:55132
I20260928 14:57:13.862038 7379 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b03be264b9d42bca5e06b6da9054922 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:57:13 UTC) from {username='oryx', principal='oryx/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:35654
I20260928 14:57:13.862372 7688 tablet_replica.cc:330] T 8b03be264b9d42bca5e06b6da9054922 P 9659e13c2242471c8899cf4f87d3fa36: stopping tablet replica
I20260928 14:57:13.862614 7688 raft_consensus.cc:2229] T 8b03be264b9d42bca5e06b6da9054922 P 9659e13c2242471c8899cf4f87d3fa36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:57:13.862504 7102 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42460:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 14:57:13.862913 7688 raft_consensus.cc:2258] T 8b03be264b9d42bca5e06b6da9054922 P 9659e13c2242471c8899cf4f87d3fa36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:57:13.863260 7687 tablet_replica.cc:330] T 8b03be264b9d42bca5e06b6da9054922 P 2a617e1e19a24cbc8c99e83995afc049: stopping tablet replica
W20260928 14:57:13.863284 7102 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:57:13.863530 7687 raft_consensus.cc:2229] T 8b03be264b9d42bca5e06b6da9054922 P 2a617e1e19a24cbc8c99e83995afc049 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:57:13.863849 7687 raft_consensus.cc:2258] T 8b03be264b9d42bca5e06b6da9054922 P 2a617e1e19a24cbc8c99e83995afc049 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:57:13.864251 7686 tablet_replica.cc:330] T 8b03be264b9d42bca5e06b6da9054922 P 48c7f448d8ae483ca98628d00e3b6fcc: stopping tablet replica
I20260928 14:57:13.864478 7686 raft_consensus.cc:2229] T 8b03be264b9d42bca5e06b6da9054922 P 48c7f448d8ae483ca98628d00e3b6fcc [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:57:13.865103 7686 raft_consensus.cc:2258] T 8b03be264b9d42bca5e06b6da9054922 P 48c7f448d8ae483ca98628d00e3b6fcc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:57:13.867053 7688 ts_tablet_manager.cc:1905] T 8b03be264b9d42bca5e06b6da9054922 P 9659e13c2242471c8899cf4f87d3fa36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:57:13.872388 7239 tablet_service.cc:1467] Processing CreateTablet for tablet 3032cf5d64ac4e2a8460df06cd37fcc2 (DEFAULT_TABLE table=test-table [id=5b23278af05243a695aa463c1fb3f333]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:57:13.872924 7239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3032cf5d64ac4e2a8460df06cd37fcc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:13.872879 7379 tablet_service.cc:1467] Processing CreateTablet for tablet 3032cf5d64ac4e2a8460df06cd37fcc2 (DEFAULT_TABLE table=test-table [id=5b23278af05243a695aa463c1fb3f333]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:57:13.873438 7379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3032cf5d64ac4e2a8460df06cd37fcc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:13.875882 7517 tablet_service.cc:1467] Processing CreateTablet for tablet 3032cf5d64ac4e2a8460df06cd37fcc2 (DEFAULT_TABLE table=test-table [id=5b23278af05243a695aa463c1fb3f333]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:57:13.876437 7517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3032cf5d64ac4e2a8460df06cd37fcc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:13.878453 7687 ts_tablet_manager.cc:1905] T 8b03be264b9d42bca5e06b6da9054922 P 2a617e1e19a24cbc8c99e83995afc049: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:57:13.883716 7703 tablet_bootstrap.cc:492] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 48c7f448d8ae483ca98628d00e3b6fcc: Bootstrap starting.
I20260928 14:57:13.885406 7686 ts_tablet_manager.cc:1905] T 8b03be264b9d42bca5e06b6da9054922 P 48c7f448d8ae483ca98628d00e3b6fcc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:57:13.886402 7703 tablet_bootstrap.cc:654] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 48c7f448d8ae483ca98628d00e3b6fcc: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:13.889102 7703 tablet_bootstrap.cc:492] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 48c7f448d8ae483ca98628d00e3b6fcc: No bootstrap required, opened a new log
I20260928 14:57:13.889273 7703 ts_tablet_manager.cc:1397] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 48c7f448d8ae483ca98628d00e3b6fcc: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 14:57:13.890105 7703 raft_consensus.cc:348] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 48c7f448d8ae483ca98628d00e3b6fcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 } } peers { permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 } } peers { permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 } }
I20260928 14:57:13.890369 7703 raft_consensus.cc:374] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 48c7f448d8ae483ca98628d00e3b6fcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:13.890444 7703 raft_consensus.cc:729] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 48c7f448d8ae483ca98628d00e3b6fcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48c7f448d8ae483ca98628d00e3b6fcc, State: Initialized, Role: FOLLOWER
I20260928 14:57:13.890700 7703 consensus_queue.cc:260] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 48c7f448d8ae483ca98628d00e3b6fcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 } } peers { permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 } } peers { permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 } }
I20260928 14:57:13.891356 7703 ts_tablet_manager.cc:1428] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 48c7f448d8ae483ca98628d00e3b6fcc: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260928 14:57:13.898078 7704 tablet_bootstrap.cc:492] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 2a617e1e19a24cbc8c99e83995afc049: Bootstrap starting.
I20260928 14:57:13.898078 7688 ts_tablet_manager.cc:1918] T 8b03be264b9d42bca5e06b6da9054922 P 9659e13c2242471c8899cf4f87d3fa36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:57:13.898195 7702 tablet_bootstrap.cc:492] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 9659e13c2242471c8899cf4f87d3fa36: Bootstrap starting.
I20260928 14:57:13.898272 7688 log.cc:1196] T 8b03be264b9d42bca5e06b6da9054922 P 9659e13c2242471c8899cf4f87d3fa36: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-0/wal/wals/8b03be264b9d42bca5e06b6da9054922
I20260928 14:57:13.898727 7688 ts_tablet_manager.cc:1939] T 8b03be264b9d42bca5e06b6da9054922 P 9659e13c2242471c8899cf4f87d3fa36: Deleting consensus metadata
I20260928 14:57:13.899981 7702 tablet_bootstrap.cc:654] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 9659e13c2242471c8899cf4f87d3fa36: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:13.900075 7704 tablet_bootstrap.cc:654] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 2a617e1e19a24cbc8c99e83995afc049: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:13.901273 7086 catalog_manager.cc:4841] TS 9659e13c2242471c8899cf4f87d3fa36 (127.31.40.1:43671): tablet 8b03be264b9d42bca5e06b6da9054922 (table test-table [id=61102a528fa04ab1b4b02846dfe476bf]) successfully deleted
I20260928 14:57:13.903815 7687 ts_tablet_manager.cc:1918] T 8b03be264b9d42bca5e06b6da9054922 P 2a617e1e19a24cbc8c99e83995afc049: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:57:13.906136 7687 log.cc:1196] T 8b03be264b9d42bca5e06b6da9054922 P 2a617e1e19a24cbc8c99e83995afc049: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-2/wal/wals/8b03be264b9d42bca5e06b6da9054922
I20260928 14:57:13.906610 7687 ts_tablet_manager.cc:1939] T 8b03be264b9d42bca5e06b6da9054922 P 2a617e1e19a24cbc8c99e83995afc049: Deleting consensus metadata
I20260928 14:57:13.907894 7086 catalog_manager.cc:4841] TS 2a617e1e19a24cbc8c99e83995afc049 (127.31.40.3:35371): tablet 8b03be264b9d42bca5e06b6da9054922 (table test-table [id=61102a528fa04ab1b4b02846dfe476bf]) successfully deleted
I20260928 14:57:13.911237 7702 tablet_bootstrap.cc:492] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 9659e13c2242471c8899cf4f87d3fa36: No bootstrap required, opened a new log
I20260928 14:57:13.911463 7702 ts_tablet_manager.cc:1397] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 9659e13c2242471c8899cf4f87d3fa36: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.009s
I20260928 14:57:13.911947 7686 ts_tablet_manager.cc:1918] T 8b03be264b9d42bca5e06b6da9054922 P 48c7f448d8ae483ca98628d00e3b6fcc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:57:13.912353 7702 raft_consensus.cc:348] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 9659e13c2242471c8899cf4f87d3fa36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 } } peers { permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 } } peers { permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 } }
I20260928 14:57:13.912635 7702 raft_consensus.cc:374] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 9659e13c2242471c8899cf4f87d3fa36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:13.912719 7702 raft_consensus.cc:729] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 9659e13c2242471c8899cf4f87d3fa36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9659e13c2242471c8899cf4f87d3fa36, State: Initialized, Role: FOLLOWER
I20260928 14:57:13.913055 7702 consensus_queue.cc:260] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 9659e13c2242471c8899cf4f87d3fa36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 } } peers { permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 } } peers { permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 } }
I20260928 14:57:13.914104 7704 tablet_bootstrap.cc:492] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 2a617e1e19a24cbc8c99e83995afc049: No bootstrap required, opened a new log
I20260928 14:57:13.914279 7704 ts_tablet_manager.cc:1397] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 2a617e1e19a24cbc8c99e83995afc049: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.007s
I20260928 14:57:13.915041 7704 raft_consensus.cc:348] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 2a617e1e19a24cbc8c99e83995afc049 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 } } peers { permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 } } peers { permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 } }
I20260928 14:57:13.915284 7704 raft_consensus.cc:374] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 2a617e1e19a24cbc8c99e83995afc049 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:13.915369 7704 raft_consensus.cc:729] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 2a617e1e19a24cbc8c99e83995afc049 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a617e1e19a24cbc8c99e83995afc049, State: Initialized, Role: FOLLOWER
I20260928 14:57:13.915652 7704 consensus_queue.cc:260] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 2a617e1e19a24cbc8c99e83995afc049 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 } } peers { permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 } } peers { permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 } }
I20260928 14:57:13.916235 7704 ts_tablet_manager.cc:1428] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 2a617e1e19a24cbc8c99e83995afc049: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:57:13.918771 7686 log.cc:1196] T 8b03be264b9d42bca5e06b6da9054922 P 48c7f448d8ae483ca98628d00e3b6fcc: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-1/wal/wals/8b03be264b9d42bca5e06b6da9054922
I20260928 14:57:13.921130 7686 ts_tablet_manager.cc:1939] T 8b03be264b9d42bca5e06b6da9054922 P 48c7f448d8ae483ca98628d00e3b6fcc: Deleting consensus metadata
I20260928 14:57:13.922580 7087 catalog_manager.cc:4841] TS 48c7f448d8ae483ca98628d00e3b6fcc (127.31.40.2:34247): tablet 8b03be264b9d42bca5e06b6da9054922 (table test-table [id=61102a528fa04ab1b4b02846dfe476bf]) successfully deleted
I20260928 14:57:13.922726 7702 ts_tablet_manager.cc:1428] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 9659e13c2242471c8899cf4f87d3fa36: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20260928 14:57:14.113605 7714 raft_consensus.cc:482] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 48c7f448d8ae483ca98628d00e3b6fcc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:57:14.113858 7714 raft_consensus.cc:504] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 48c7f448d8ae483ca98628d00e3b6fcc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 } } peers { permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 } } peers { permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 } }
I20260928 14:57:14.114691 7714 leader_election.cc:288] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 48c7f448d8ae483ca98628d00e3b6fcc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9659e13c2242471c8899cf4f87d3fa36 (127.31.40.1:43671), 2a617e1e19a24cbc8c99e83995afc049 (127.31.40.3:35371)
I20260928 14:57:14.115360 7258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3032cf5d64ac4e2a8460df06cd37fcc2" candidate_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9659e13c2242471c8899cf4f87d3fa36" is_pre_election: true
I20260928 14:57:14.115401 7538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3032cf5d64ac4e2a8460df06cd37fcc2" candidate_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a617e1e19a24cbc8c99e83995afc049" is_pre_election: true
I20260928 14:57:14.115633 7258 raft_consensus.cc:2454] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 9659e13c2242471c8899cf4f87d3fa36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48c7f448d8ae483ca98628d00e3b6fcc in term 0.
I20260928 14:57:14.115687 7538 raft_consensus.cc:2454] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 2a617e1e19a24cbc8c99e83995afc049 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48c7f448d8ae483ca98628d00e3b6fcc in term 0.
I20260928 14:57:14.116111 7329 leader_election.cc:302] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 48c7f448d8ae483ca98628d00e3b6fcc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48c7f448d8ae483ca98628d00e3b6fcc, 9659e13c2242471c8899cf4f87d3fa36; no voters:
I20260928 14:57:14.116402 7714 raft_consensus.cc:2781] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 48c7f448d8ae483ca98628d00e3b6fcc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:57:14.116547 7714 raft_consensus.cc:482] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 48c7f448d8ae483ca98628d00e3b6fcc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:57:14.116695 7714 raft_consensus.cc:3037] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 48c7f448d8ae483ca98628d00e3b6fcc [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:14.119184 7714 raft_consensus.cc:504] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 48c7f448d8ae483ca98628d00e3b6fcc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 } } peers { permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 } } peers { permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 } }
I20260928 14:57:14.119834 7714 leader_election.cc:288] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 48c7f448d8ae483ca98628d00e3b6fcc [CANDIDATE]: Term 1 election: Requested vote from peers 9659e13c2242471c8899cf4f87d3fa36 (127.31.40.1:43671), 2a617e1e19a24cbc8c99e83995afc049 (127.31.40.3:35371)
I20260928 14:57:14.120414 7538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3032cf5d64ac4e2a8460df06cd37fcc2" candidate_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a617e1e19a24cbc8c99e83995afc049"
I20260928 14:57:14.120437 7258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3032cf5d64ac4e2a8460df06cd37fcc2" candidate_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9659e13c2242471c8899cf4f87d3fa36"
I20260928 14:57:14.120644 7538 raft_consensus.cc:3037] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 2a617e1e19a24cbc8c99e83995afc049 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:14.120685 7258 raft_consensus.cc:3037] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 9659e13c2242471c8899cf4f87d3fa36 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:14.123461 7538 raft_consensus.cc:2454] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 2a617e1e19a24cbc8c99e83995afc049 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48c7f448d8ae483ca98628d00e3b6fcc in term 1.
I20260928 14:57:14.123952 7258 raft_consensus.cc:2454] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 9659e13c2242471c8899cf4f87d3fa36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48c7f448d8ae483ca98628d00e3b6fcc in term 1.
I20260928 14:57:14.124015 7329 leader_election.cc:302] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 48c7f448d8ae483ca98628d00e3b6fcc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a617e1e19a24cbc8c99e83995afc049, 48c7f448d8ae483ca98628d00e3b6fcc; no voters:
I20260928 14:57:14.124358 7714 raft_consensus.cc:2781] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 48c7f448d8ae483ca98628d00e3b6fcc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:57:14.124553 7714 raft_consensus.cc:686] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 48c7f448d8ae483ca98628d00e3b6fcc [term 1 LEADER]: Becoming Leader. State: Replica: 48c7f448d8ae483ca98628d00e3b6fcc, State: Running, Role: LEADER
I20260928 14:57:14.124851 7714 consensus_queue.cc:237] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 48c7f448d8ae483ca98628d00e3b6fcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 } } peers { permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 } } peers { permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 } }
I20260928 14:57:14.127743 7102 catalog_manager.cc:5495] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 48c7f448d8ae483ca98628d00e3b6fcc reported cstate change: term changed from 0 to 1, leader changed from <none> to 48c7f448d8ae483ca98628d00e3b6fcc (127.31.40.2). New cstate: current_term: 1 leader_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48c7f448d8ae483ca98628d00e3b6fcc" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 } health_report { overall_health: UNKNOWN } } }
I20260928 14:57:14.173790 7723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:14.160953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:39839 (local address 127.31.40.3:35371)
0928 14:57:14.161463 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:14.161467 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:14.168558 (+ 7091us) server_negotiation.cc:407] Connection header received
0928 14:57:14.168799 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:14.168805 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:14.168883 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:14.168988 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:57:14.169576 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:14.170344 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:14.171858 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:14.172333 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:14.172472 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:14.172961 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:57:14.173065 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:57:14.173179 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:14.173339 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 14:57:14.173417 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":293,"threads_started":1}
I20260928 14:57:14.173856 7724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:14.160978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.3:35371 (local address 127.31.40.2:39839)
0928 14:57:14.168430 (+ 7452us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:14.168451 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:14.168649 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:14.169007 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:14.169011 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:14.169023 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:57:14.169413 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:14.169423 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:14.170500 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:14.170505 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:14.171704 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:14.171720 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:14.172533 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:14.172549 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:14.172692 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:14.172753 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:57:14.173180 (+ 427us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:57:14.173193 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 14:57:14.173306 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 14:57:14.173450 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7259,"thread_start_us":7095,"threads_started":1}
I20260928 14:57:14.185696 7538 raft_consensus.cc:1261] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 2a617e1e19a24cbc8c99e83995afc049 [term 1 FOLLOWER]: Refusing update from remote peer 48c7f448d8ae483ca98628d00e3b6fcc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:57:14.186232 7714 consensus_queue.cc:1035] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 48c7f448d8ae483ca98628d00e3b6fcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a617e1e19a24cbc8c99e83995afc049" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 35371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:57:14.186396 7258 raft_consensus.cc:1261] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 9659e13c2242471c8899cf4f87d3fa36 [term 1 FOLLOWER]: Refusing update from remote peer 48c7f448d8ae483ca98628d00e3b6fcc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:57:14.186964 7714 consensus_queue.cc:1035] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 48c7f448d8ae483ca98628d00e3b6fcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9659e13c2242471c8899cf4f87d3fa36" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:57:14.218884 7711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:14.213134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:50639 (local address 127.31.40.2:34247)
0928 14:57:14.213335 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:14.213341 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:14.213568 (+ 227us) server_negotiation.cc:407] Connection header received
0928 14:57:14.213756 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:14.213759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:14.213819 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:14.213892 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:57:14.214390 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:14.215459 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:14.217103 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:14.217568 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:14.217711 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:14.218126 (+ 415us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:57:14.218260 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:57:14.218347 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:14.218494 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 14:57:14.218570 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 14:57:14.218948 7730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:14.212853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.2:34247 (local address 127.31.40.3:50639)
0928 14:57:14.213461 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:14.213484 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:14.213636 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:14.213943 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:14.213949 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:14.213964 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:57:14.214224 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:14.214240 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:14.215588 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:14.215594 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:14.216964 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:14.216975 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:14.217725 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:14.217731 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:14.217886 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:14.217923 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:57:14.218348 (+ 425us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:57:14.218362 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 14:57:14.218473 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 14:57:14.218611 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":286,"threads_started":1}
I20260928 14:57:14.252033 7102 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:57:14.252301 7102 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:57:14.255530 7102 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 73109dfa9b4d4463a096a15f0501a28c: Sending DeleteTablet for 3 replicas of tablet 3032cf5d64ac4e2a8460df06cd37fcc2
I20260928 14:57:14.256467 7239 tablet_service.cc:1514] Processing DeleteTablet for tablet 3032cf5d64ac4e2a8460df06cd37fcc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:57:14 UTC) from {username='oryx', principal='oryx/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:51354
I20260928 14:57:14.256554 7379 tablet_service.cc:1514] Processing DeleteTablet for tablet 3032cf5d64ac4e2a8460df06cd37fcc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:57:14 UTC) from {username='oryx', principal='oryx/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:35654
I20260928 14:57:14.257014 7688 tablet_replica.cc:330] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 9659e13c2242471c8899cf4f87d3fa36: stopping tablet replica
I20260928 14:57:14.257028 7517 tablet_service.cc:1514] Processing DeleteTablet for tablet 3032cf5d64ac4e2a8460df06cd37fcc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:57:14 UTC) from {username='oryx', principal='oryx/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:55132
I20260928 14:57:14.257226 7688 raft_consensus.cc:2229] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 9659e13c2242471c8899cf4f87d3fa36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:57:14.257339 7687 tablet_replica.cc:330] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 2a617e1e19a24cbc8c99e83995afc049: stopping tablet replica
I20260928 14:57:14.257429 7688 raft_consensus.cc:2258] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 9659e13c2242471c8899cf4f87d3fa36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:57:14.257520 7687 raft_consensus.cc:2229] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 2a617e1e19a24cbc8c99e83995afc049 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:57:14.257725 7687 raft_consensus.cc:2258] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 2a617e1e19a24cbc8c99e83995afc049 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:57:14.257735 7686 tablet_replica.cc:330] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 48c7f448d8ae483ca98628d00e3b6fcc: stopping tablet replica
I20260928 14:57:14.258036 7686 raft_consensus.cc:2229] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 48c7f448d8ae483ca98628d00e3b6fcc [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:57:14.258407 7686 raft_consensus.cc:2258] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 48c7f448d8ae483ca98628d00e3b6fcc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:57:14.258894 7687 ts_tablet_manager.cc:1905] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 2a617e1e19a24cbc8c99e83995afc049: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:57:14.259724 7688 ts_tablet_manager.cc:1905] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 9659e13c2242471c8899cf4f87d3fa36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:57:14.260687 7686 ts_tablet_manager.cc:1905] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 48c7f448d8ae483ca98628d00e3b6fcc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:57:14.266328 7687 ts_tablet_manager.cc:1918] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 2a617e1e19a24cbc8c99e83995afc049: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:57:14.266479 7687 log.cc:1196] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 2a617e1e19a24cbc8c99e83995afc049: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-2/wal/wals/3032cf5d64ac4e2a8460df06cd37fcc2
I20260928 14:57:14.266953 7687 ts_tablet_manager.cc:1939] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 2a617e1e19a24cbc8c99e83995afc049: Deleting consensus metadata
I20260928 14:57:14.268180 7086 catalog_manager.cc:4841] TS 2a617e1e19a24cbc8c99e83995afc049 (127.31.40.3:35371): tablet 3032cf5d64ac4e2a8460df06cd37fcc2 (table test-table [id=5b23278af05243a695aa463c1fb3f333]) successfully deleted
I20260928 14:57:14.269557 7688 ts_tablet_manager.cc:1918] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 9659e13c2242471c8899cf4f87d3fa36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:57:14.269761 7688 log.cc:1196] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 9659e13c2242471c8899cf4f87d3fa36: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-0/wal/wals/3032cf5d64ac4e2a8460df06cd37fcc2
I20260928 14:57:14.270308 7688 ts_tablet_manager.cc:1939] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 9659e13c2242471c8899cf4f87d3fa36: Deleting consensus metadata
I20260928 14:57:14.271543 7086 catalog_manager.cc:4841] TS 9659e13c2242471c8899cf4f87d3fa36 (127.31.40.1:43671): tablet 3032cf5d64ac4e2a8460df06cd37fcc2 (table test-table [id=5b23278af05243a695aa463c1fb3f333]) successfully deleted
I20260928 14:57:14.276566 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 7173
I20260928 14:57:14.276983 7686 ts_tablet_manager.cc:1918] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 48c7f448d8ae483ca98628d00e3b6fcc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:57:14.277251 7686 log.cc:1196] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 48c7f448d8ae483ca98628d00e3b6fcc: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607365362981-31904-0/minicluster-data/ts-1/wal/wals/3032cf5d64ac4e2a8460df06cd37fcc2
I20260928 14:57:14.277823 7686 ts_tablet_manager.cc:1939] T 3032cf5d64ac4e2a8460df06cd37fcc2 P 48c7f448d8ae483ca98628d00e3b6fcc: Deleting consensus metadata
I20260928 14:57:14.279564 7087 catalog_manager.cc:4841] TS 48c7f448d8ae483ca98628d00e3b6fcc (127.31.40.2:34247): tablet 3032cf5d64ac4e2a8460df06cd37fcc2 (table test-table [id=5b23278af05243a695aa463c1fb3f333]) successfully deleted
Sep 28 14:57:14 dist-test-slave-j968 krb5kdc[7057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.40.1@KRBTEST.COM
I20260928 14:57:14.306362 7733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:14.278509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51384 (local address 127.31.40.1:43671)
0928 14:57:14.280384 (+ 1875us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:14.280392 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:14.280867 (+ 475us) server_negotiation.cc:407] Connection header received
0928 14:57:14.281055 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:14.281062 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:14.281158 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:14.281281 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:14.282953 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:14.284264 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:14.286089 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:14.286854 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:14.287018 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:14.297907 (+ 10889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:14.297930 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:14.297954 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:14.298020 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:14.302414 (+ 4394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:14.303418 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:14.303429 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:14.303433 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:14.303638 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:14.304174 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:14.304180 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:14.304184 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:14.304733 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:14.305151 (+ 418us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:14.305472 (+ 321us) server_negotiation.cc:299] Negotiation successful
0928 14:57:14.305731 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1733,"thread_start_us":1597,"threads_started":1}
I20260928 14:57:14.307109 7304 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:57:14.444768 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 7173
I20260928 14:57:14.470232 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 7313
Sep 28 14:57:14 dist-test-slave-j968 krb5kdc[7057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.40.2@KRBTEST.COM
I20260928 14:57:14.491567 7711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:14.471487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35658 (local address 127.31.40.2:34247)
0928 14:57:14.471896 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:14.471902 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:14.471921 (+ 19us) server_negotiation.cc:407] Connection header received
0928 14:57:14.471990 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:14.471996 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:14.472076 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:14.472200 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:14.474360 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:14.475627 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:14.477868 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:14.478450 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:14.478593 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:14.483471 (+ 4878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:14.483516 (+ 45us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:14.483545 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:14.483599 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:14.487945 (+ 4346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:14.488867 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:14.488877 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:14.488881 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:14.489038 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:14.489588 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:14.489594 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:14.489597 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:14.490140 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:14.490505 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:14.490926 (+ 421us) server_negotiation.cc:299] Negotiation successful
0928 14:57:14.491176 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237}
I20260928 14:57:14.492357 7444 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:57:14.637106 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 7313
I20260928 14:57:14.664430 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 7453
Sep 28 14:57:14 dist-test-slave-j968 krb5kdc[7057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.40.3@KRBTEST.COM
I20260928 14:57:14.677022 7736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:14.675856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:43671 (local address 127.31.40.3:57685)
0928 14:57:14.676582 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:14.676744 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.1:43671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":290,"threads_started":1}
W20260928 14:57:14.677379 7469 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.40.1:43671: connect: Connection refused (error 111)
I20260928 14:57:14.679119 7736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:14.678685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:43671 (local address 127.31.40.3:43761)
0928 14:57:14.678910 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:14.678974 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.1:43671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
W20260928 14:57:14.679301 7469 consensus_peers.cc:487] T 1ec7a1258ca6451d824026fe65c72b66 P 2a617e1e19a24cbc8c99e83995afc049 -> Peer 9659e13c2242471c8899cf4f87d3fa36 (127.31.40.1:43671): Couldn't send request to peer 9659e13c2242471c8899cf4f87d3fa36. Status: Network error: Client connection negotiation failed: client connection to 127.31.40.1:43671: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 14:57:14.682088 7736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:14.681713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.2:34247 (local address 127.31.40.3:60009)
0928 14:57:14.681871 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:14.681951 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.2:34247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 14:57:14.682685 7723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:14.665307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55144 (local address 127.31.40.3:35371)
0928 14:57:14.665538 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:14.665545 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:14.665679 (+ 134us) server_negotiation.cc:407] Connection header received
0928 14:57:14.666014 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:14.666020 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:14.666112 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:14.666233 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:14.667635 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:14.668591 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:14.669961 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:14.670441 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:14.670597 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:14.675653 (+ 5056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:14.675676 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:14.675695 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:14.675733 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:14.679730 (+ 3997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:14.680516 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:14.680524 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:14.680526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:14.680667 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:14.681053 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:14.681057 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:14.681059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:14.681435 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:14.681821 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:14.682103 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 14:57:14.682352 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 14:57:14.683157 7736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:14.682831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.2:34247 (local address 127.31.40.3:56539)
0928 14:57:14.682976 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:14.683054 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.2:34247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 14:57:14.683342 7470 consensus_peers.cc:487] T 1ec7a1258ca6451d824026fe65c72b66 P 2a617e1e19a24cbc8c99e83995afc049 -> Peer 48c7f448d8ae483ca98628d00e3b6fcc (127.31.40.2:34247): Couldn't send request to peer 48c7f448d8ae483ca98628d00e3b6fcc. Status: Network error: Client connection negotiation failed: client connection to 127.31.40.2:34247: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 14:57:14.683506 7583 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:57:14.807614 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 7453
I20260928 14:57:14.831349 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 7070
I20260928 14:57:14.832696 7142 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:57:14.951373 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 7070
2026-09-28T14:57:14Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6149 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607365362981-31904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:57:15 dist-test-slave-j968 krb5kdc[7742](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:57:15 dist-test-slave-j968 krb5kdc[7742](info): set up 2 sockets
Sep 28 14:57:15 dist-test-slave-j968 krb5kdc[7742](info): commencing operation
krb5kdc: starting...
W20260928 14:57:16.077740 31904 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Sep 28 14:57:16 dist-test-slave-j968 krb5kdc[7742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:57:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:57:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab.
Entry for principal kudu/127.31.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab.
I20260928 14:57:16.316737 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607365362981-31904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607365362981-31904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607365362981-31904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:33431
--webserver_interface=127.31.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:35159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.40.62:33431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:57:16.649283 7755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:16.649627 7755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:16.649757 7755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:16.657550 7755 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:57:16.657680 7755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:16.657763 7755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:16.657879 7755 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:57:16.657975 7755 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:57:16.669109 7755 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:35159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607365362981-31904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607365362981-31904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.40.62:33431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:33431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607365362981-31904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:57:16.670529 7755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:16.671991 7755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:16.685216 7760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:16.685189 7763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:16.685266 7761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:16.685746 7755 server_base.cc:1034] running on GCE node
I20260928 14:57:16.686815 7755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:16.688696 7755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:16.689990 7755 hybrid_clock.cc:648] HybridClock initialized: now 1790607436689945 us; error 38 us; skew 500 ppm
I20260928 14:57:16.690411 7755 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T14:57:16Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1802 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:57:16 dist-test-slave-j968 krb5kdc[7769](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:57:16 dist-test-slave-j968 krb5kdc[7769](info): set up 2 sockets
Sep 28 14:57:16 dist-test-slave-j968 krb5kdc[7769](info): commencing operation
krb5kdc: starting...
W20260928 14:57:17.862097 31904 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Sep 28 14:57:18 dist-test-slave-j968 krb5kdc[7769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:57:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:57:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab.
Entry for principal kudu/127.31.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab.
I20260928 14:57:18.099509 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:40335
--webserver_interface=127.31.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:35391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.40.62:40335
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:57:18.443470 7782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:18.443842 7782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:18.444046 7782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:18.451674 7782 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:57:18.451853 7782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:18.452006 7782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:18.452113 7782 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:57:18.452198 7782 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:57:18.462736 7782 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:35391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.40.62:40335
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:40335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:57:18.464283 7782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:18.465658 7782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:18.477979 7787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:18.478006 7788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:18.478063 7790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:18.479151 7782 server_base.cc:1034] running on GCE node
I20260928 14:57:18.480068 7782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:18.481537 7782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:18.482756 7782 hybrid_clock.cc:648] HybridClock initialized: now 1790607438482687 us; error 51 us; skew 500 ppm
I20260928 14:57:18.483180 7782 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:57:18 dist-test-slave-j968 krb5kdc[7769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607438, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:18.490394 7782 init.cc:377] Logged in from keytab as kudu/127.31.40.62@KRBTEST.COM (short username kudu)
I20260928 14:57:18.492061 7782 webserver.cc:460] Webserver started at http://127.31.40.62:37203/ using document root <none> and password file <none>
I20260928 14:57:18.492810 7782 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:18.492971 7782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:18.493296 7782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:18.496459 7782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/master-0/data/instance:
uuid: "136d295a29b14ad6a27ff1829a44c665"
format_stamp: "Formatted at 2026-09-28 14:57:18 on dist-test-slave-j968"
server_key: "9b97c69ca204bc98cd4a5329c6e4a208"
server_key_iv: "359185d4458e28c647d25565291fe42f"
server_key_version: "encryptionkey@0"
I20260928 14:57:18.497236 7782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/master-0/wal/instance:
uuid: "136d295a29b14ad6a27ff1829a44c665"
format_stamp: "Formatted at 2026-09-28 14:57:18 on dist-test-slave-j968"
server_key: "9b97c69ca204bc98cd4a5329c6e4a208"
server_key_iv: "359185d4458e28c647d25565291fe42f"
server_key_version: "encryptionkey@0"
I20260928 14:57:18.502751 7782 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 14:57:18.506631 7797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:18.508116 7782 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:57:18.508306 7782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/master-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/master-0/wal
uuid: "136d295a29b14ad6a27ff1829a44c665"
format_stamp: "Formatted at 2026-09-28 14:57:18 on dist-test-slave-j968"
server_key: "9b97c69ca204bc98cd4a5329c6e4a208"
server_key_iv: "359185d4458e28c647d25565291fe42f"
server_key_version: "encryptionkey@0"
I20260928 14:57:18.508545 7782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:18.537125 7782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:18.542225 7782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:18.542506 7782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:18.562799 7782 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.62:40335
I20260928 14:57:18.562839 7848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.62:40335 every 8 connection(s)
I20260928 14:57:18.564410 7782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
I20260928 14:57:18.564805 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 7782
I20260928 14:57:18.565163 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/master-0/wal/instance
I20260928 14:57:18.566069 31904 external_mini_cluster.cc:1428] Setting key b1bdecb6882e96b2e7607903ecce8822
I20260928 14:57:18.569579 7849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 14:57:18 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:57:18.584154 7849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 136d295a29b14ad6a27ff1829a44c665: Bootstrap starting.
I20260928 14:57:18.587863 7849 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 136d295a29b14ad6a27ff1829a44c665: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:18.589335 7849 log.cc:824] T 00000000000000000000000000000000 P 136d295a29b14ad6a27ff1829a44c665: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:18.592487 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:18.568465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41500 (local address 127.31.40.62:40335)
0928 14:57:18.569337 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:18.569355 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:18.569412 (+ 57us) server_negotiation.cc:407] Connection header received
0928 14:57:18.572108 (+ 2696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:18.572140 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:18.572664 (+ 524us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:18.573208 (+ 544us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:18.574591 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:18.577051 (+ 2460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:18.578406 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:18.579590 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:18.579792 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:18.584154 (+ 4362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:18.584164 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:18.584180 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:18.584221 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:18.587431 (+ 3210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:18.588481 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:18.588488 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:18.588495 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:18.588695 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:18.589170 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:18.589173 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:18.589175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:18.589786 (+ 611us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:18.590170 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:18.590666 (+ 496us) server_negotiation.cc:299] Negotiation successful
0928 14:57:18.591126 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":355,"threads_started":1}
I20260928 14:57:18.593047 7849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 136d295a29b14ad6a27ff1829a44c665: No bootstrap required, opened a new log
I20260928 14:57:18.600999 7849 raft_consensus.cc:348] T 00000000000000000000000000000000 P 136d295a29b14ad6a27ff1829a44c665 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "136d295a29b14ad6a27ff1829a44c665" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 40335 } }
I20260928 14:57:18.601364 7849 raft_consensus.cc:374] T 00000000000000000000000000000000 P 136d295a29b14ad6a27ff1829a44c665 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:18.601437 7849 raft_consensus.cc:729] T 00000000000000000000000000000000 P 136d295a29b14ad6a27ff1829a44c665 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 136d295a29b14ad6a27ff1829a44c665, State: Initialized, Role: FOLLOWER
I20260928 14:57:18.602015 7849 consensus_queue.cc:260] T 00000000000000000000000000000000 P 136d295a29b14ad6a27ff1829a44c665 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "136d295a29b14ad6a27ff1829a44c665" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 40335 } }
I20260928 14:57:18.602217 7849 raft_consensus.cc:388] T 00000000000000000000000000000000 P 136d295a29b14ad6a27ff1829a44c665 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:57:18.602316 7849 raft_consensus.cc:482] T 00000000000000000000000000000000 P 136d295a29b14ad6a27ff1829a44c665 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:57:18.602439 7849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 136d295a29b14ad6a27ff1829a44c665 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:18.604717 7849 raft_consensus.cc:504] T 00000000000000000000000000000000 P 136d295a29b14ad6a27ff1829a44c665 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "136d295a29b14ad6a27ff1829a44c665" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 40335 } }
I20260928 14:57:18.605368 7849 leader_election.cc:302] T 00000000000000000000000000000000 P 136d295a29b14ad6a27ff1829a44c665 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 136d295a29b14ad6a27ff1829a44c665; no voters:
I20260928 14:57:18.605975 7849 leader_election.cc:288] T 00000000000000000000000000000000 P 136d295a29b14ad6a27ff1829a44c665 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:57:18.606158 7854 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 136d295a29b14ad6a27ff1829a44c665 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:57:18.607260 7854 raft_consensus.cc:686] T 00000000000000000000000000000000 P 136d295a29b14ad6a27ff1829a44c665 [term 1 LEADER]: Becoming Leader. State: Replica: 136d295a29b14ad6a27ff1829a44c665, State: Running, Role: LEADER
I20260928 14:57:18.607796 7854 consensus_queue.cc:237] T 00000000000000000000000000000000 P 136d295a29b14ad6a27ff1829a44c665 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "136d295a29b14ad6a27ff1829a44c665" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 40335 } }
I20260928 14:57:18.608161 7849 sys_catalog.cc:564] T 00000000000000000000000000000000 P 136d295a29b14ad6a27ff1829a44c665 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:57:18.611228 7856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 136d295a29b14ad6a27ff1829a44c665 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 136d295a29b14ad6a27ff1829a44c665. Latest consensus state: current_term: 1 leader_uuid: "136d295a29b14ad6a27ff1829a44c665" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "136d295a29b14ad6a27ff1829a44c665" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 40335 } } }
I20260928 14:57:18.611207 7855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 136d295a29b14ad6a27ff1829a44c665 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "136d295a29b14ad6a27ff1829a44c665" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "136d295a29b14ad6a27ff1829a44c665" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 40335 } } }
I20260928 14:57:18.611605 7855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 136d295a29b14ad6a27ff1829a44c665 [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:18.611604 7856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 136d295a29b14ad6a27ff1829a44c665 [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:18.613639 7861 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:57:18.619048 7861 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:57:18.628877 7861 catalog_manager.cc:1347] Generated new cluster ID: 394b3c78648c4d7da553d3d717d140df
I20260928 14:57:18.629011 7861 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:57:18.648768 7861 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:57:18.650779 7861 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:57:18.661656 7861 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 136d295a29b14ad6a27ff1829a44c665: Generated new TSK 0
I20260928 14:57:18.662869 7861 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.40.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab.
Entry for principal kudu/127.31.40.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab.
I20260928 14:57:18.725741 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.1:0
--local_ip_for_outbound_sockets=127.31.40.1
--webserver_interface=127.31.40.1
--webserver_port=0
--tserver_master_addrs=127.31.40.62:40335
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab
--principal=kudu/127.31.40.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:35391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:57:19.067709 7875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:19.068116 7875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:19.068305 7875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:19.075819 7875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:19.076040 7875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:19.076236 7875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.1
I20260928 14:57:19.086301 7875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:35391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab
--principal=kudu/127.31.40.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.40.1
--webserver_port=0
--tserver_master_addrs=127.31.40.62:40335
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.1
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:57:19.087646 7875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:19.089159 7875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:19.102474 7880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:19.102463 7881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:19.102450 7883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:19.103710 7875 server_base.cc:1034] running on GCE node
I20260928 14:57:19.104431 7875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:19.105520 7875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:19.106760 7875 hybrid_clock.cc:648] HybridClock initialized: now 1790607439106712 us; error 40 us; skew 500 ppm
I20260928 14:57:19.107088 7875 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:57:19 dist-test-slave-j968 krb5kdc[7769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607439, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:19.110764 7875 init.cc:377] Logged in from keytab as kudu/127.31.40.1@KRBTEST.COM (short username kudu)
I20260928 14:57:19.112450 7875 webserver.cc:460] Webserver started at http://127.31.40.1:42861/ using document root <none> and password file <none>
I20260928 14:57:19.113188 7875 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:19.113291 7875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:19.113556 7875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:19.116557 7875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-0/data/instance:
uuid: "2528937497ef49e6a5cc027a5d446e46"
format_stamp: "Formatted at 2026-09-28 14:57:19 on dist-test-slave-j968"
server_key: "f62746b7954c96dcc21e4c232af96918"
server_key_iv: "5b67e09400fd698df1abab626eab5ecc"
server_key_version: "encryptionkey@0"
I20260928 14:57:19.117368 7875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-0/wal/instance:
uuid: "2528937497ef49e6a5cc027a5d446e46"
format_stamp: "Formatted at 2026-09-28 14:57:19 on dist-test-slave-j968"
server_key: "f62746b7954c96dcc21e4c232af96918"
server_key_iv: "5b67e09400fd698df1abab626eab5ecc"
server_key_version: "encryptionkey@0"
I20260928 14:57:19.122820 7875 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 14:57:19.126479 7890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:19.128068 7875 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:57:19.128249 7875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-0/wal
uuid: "2528937497ef49e6a5cc027a5d446e46"
format_stamp: "Formatted at 2026-09-28 14:57:19 on dist-test-slave-j968"
server_key: "f62746b7954c96dcc21e4c232af96918"
server_key_iv: "5b67e09400fd698df1abab626eab5ecc"
server_key_version: "encryptionkey@0"
I20260928 14:57:19.128500 7875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:19.158387 7875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:19.163014 7875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:19.163332 7875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:19.164464 7875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:57:19.165861 7875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:57:19.165935 7875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:19.166111 7875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:57:19.166167 7875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:19.201097 7875 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.1:46017
I20260928 14:57:19.201197 8002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.1:46017 every 8 connection(s)
I20260928 14:57:19.202538 7875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
I20260928 14:57:19.209163 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 7875
I20260928 14:57:19.209396 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-0/wal/instance
I20260928 14:57:19.210181 31904 external_mini_cluster.cc:1428] Setting key dc0d6c9dbf66bcf6e834660900d34332
Sep 28 14:57:19 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607439, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.1@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:57:19.224182 8005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:19.205339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:43267 (local address 127.31.40.62:40335)
0928 14:57:19.205811 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:19.205815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:19.206718 (+ 903us) server_negotiation.cc:407] Connection header received
0928 14:57:19.208801 (+ 2083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:19.208807 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:19.208902 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:19.209027 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:19.210799 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:19.211784 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:19.213587 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:19.214005 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:19.214148 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:19.218171 (+ 4023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:19.218177 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:19.218181 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:19.218215 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:19.220942 (+ 2727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:19.221756 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:19.221762 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:19.221765 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:19.221896 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:19.222341 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:19.222346 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:19.222350 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:19.222638 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:19.222797 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:19.223547 (+ 750us) server_negotiation.cc:299] Negotiation successful
0928 14:57:19.223775 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":258,"threads_started":1}
I20260928 14:57:19.225189 8006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:19.205854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40335 (local address 127.31.40.1:43267)
0928 14:57:19.206547 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:19.206592 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:19.208526 (+ 1934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:19.209271 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:19.209280 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:19.209968 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:19.210625 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:19.210644 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:19.212019 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:19.212024 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:19.213437 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:19.213450 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:19.214246 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:19.214255 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:19.214500 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:19.215279 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:19.215303 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:19.217908 (+ 2605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:19.221105 (+ 3197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:19.221115 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:19.221134 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:19.221609 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:19.222019 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:19.222024 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:19.222028 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:19.222220 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:19.222770 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:19.222778 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:19.223157 (+ 379us) client_negotiation.cc:769] Sending connection context
0928 14:57:19.223435 (+ 278us) client_negotiation.cc:240] Negotiation successful
0928 14:57:19.224024 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":278,"threads_started":1}
I20260928 14:57:19.226852 8003 heartbeater.cc:347] Connected to a master server at 127.31.40.62:40335
I20260928 14:57:19.227183 8003 heartbeater.cc:464] Registering TS with master...
I20260928 14:57:19.227963 8003 heartbeater.cc:511] Master 127.31.40.62:40335 requested a full tablet report, sending...
I20260928 14:57:19.230077 7814 ts_manager.cc:194] Registered new tserver with Master: 2528937497ef49e6a5cc027a5d446e46 (127.31.40.1:46017)
I20260928 14:57:19.231549 7814 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.1@KRBTEST.COM'} at 127.31.40.1:43267
WARNING: no policy specified for kudu/127.31.40.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab.
Entry for principal kudu/127.31.40.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab.
I20260928 14:57:19.262596 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.2:0
--local_ip_for_outbound_sockets=127.31.40.2
--webserver_interface=127.31.40.2
--webserver_port=0
--tserver_master_addrs=127.31.40.62:40335
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab
--principal=kudu/127.31.40.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:35391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:57:19.599182 8009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:19.599467 8009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:19.599637 8009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:19.606997 8009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:19.607123 8009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:19.607308 8009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.2
I20260928 14:57:19.617779 8009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:35391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab
--principal=kudu/127.31.40.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.40.2
--webserver_port=0
--tserver_master_addrs=127.31.40.62:40335
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.2
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:57:19.619107 8009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:19.620469 8009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:19.634322 8014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:19.634286 8017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:19.634270 8015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:19.635227 8009 server_base.cc:1034] running on GCE node
I20260928 14:57:19.635833 8009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:19.636905 8009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:19.638139 8009 hybrid_clock.cc:648] HybridClock initialized: now 1790607439638090 us; error 38 us; skew 500 ppm
I20260928 14:57:19.638444 8009 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:57:19 dist-test-slave-j968 krb5kdc[7769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607439, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:19.641855 8009 init.cc:377] Logged in from keytab as kudu/127.31.40.2@KRBTEST.COM (short username kudu)
I20260928 14:57:19.643353 8009 webserver.cc:460] Webserver started at http://127.31.40.2:36029/ using document root <none> and password file <none>
I20260928 14:57:19.644135 8009 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:19.644243 8009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:19.644498 8009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:19.647500 8009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-1/data/instance:
uuid: "9585a49fcac94e64bb952314a7ccbbe3"
format_stamp: "Formatted at 2026-09-28 14:57:19 on dist-test-slave-j968"
server_key: "80e1ca0851efcc32e5bd35e8194f6913"
server_key_iv: "dc5aa5b1a64f93680d017522c6516101"
server_key_version: "encryptionkey@0"
I20260928 14:57:19.648403 8009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-1/wal/instance:
uuid: "9585a49fcac94e64bb952314a7ccbbe3"
format_stamp: "Formatted at 2026-09-28 14:57:19 on dist-test-slave-j968"
server_key: "80e1ca0851efcc32e5bd35e8194f6913"
server_key_iv: "dc5aa5b1a64f93680d017522c6516101"
server_key_version: "encryptionkey@0"
I20260928 14:57:19.653795 8009 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 14:57:19.658269 8024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:19.660108 8009 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20260928 14:57:19.660284 8009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-1/wal
uuid: "9585a49fcac94e64bb952314a7ccbbe3"
format_stamp: "Formatted at 2026-09-28 14:57:19 on dist-test-slave-j968"
server_key: "80e1ca0851efcc32e5bd35e8194f6913"
server_key_iv: "dc5aa5b1a64f93680d017522c6516101"
server_key_version: "encryptionkey@0"
I20260928 14:57:19.660506 8009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:19.683177 8009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:19.687759 8009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:19.688069 8009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:19.689167 8009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:57:19.690642 8009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:57:19.690716 8009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:19.690858 8009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:57:19.690910 8009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:19.726938 8009 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.2:38273
I20260928 14:57:19.727041 8136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.2:38273 every 8 connection(s)
I20260928 14:57:19.728437 8009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
I20260928 14:57:19.736219 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 8009
I20260928 14:57:19.736430 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-1/wal/instance
I20260928 14:57:19.736989 31904 external_mini_cluster.cc:1428] Setting key aacbe0227bc5e618cf971fc233654339
Sep 28 14:57:19 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607439, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.2@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:57:19.751240 8139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:19.731210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:57559 (local address 127.31.40.62:40335)
0928 14:57:19.731689 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:19.731693 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:19.732796 (+ 1103us) server_negotiation.cc:407] Connection header received
0928 14:57:19.734876 (+ 2080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:19.734881 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:19.734980 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:19.735111 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:19.737256 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:19.737988 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:19.740500 (+ 2512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:19.740932 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:19.741032 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:19.745402 (+ 4370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:19.745408 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:19.745411 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:19.745445 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:19.748118 (+ 2673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:19.748899 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:19.748909 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:19.748911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:19.749034 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:19.749465 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:19.749470 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:19.749473 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:19.749744 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:19.749894 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:19.750617 (+ 723us) server_negotiation.cc:299] Negotiation successful
0928 14:57:19.750846 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":270,"threads_started":1}
I20260928 14:57:19.752317 8140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:19.731769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40335 (local address 127.31.40.2:57559)
0928 14:57:19.732628 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:19.732675 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:19.734574 (+ 1899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:19.735377 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:19.735385 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:19.736130 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:19.737040 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:19.737076 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:19.738221 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:19.738225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:19.740347 (+ 2122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:19.740364 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:19.741181 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:19.741190 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:19.741542 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:19.742335 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:19.742367 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:19.745123 (+ 2756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:19.748267 (+ 3144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:19.748277 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:19.748297 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:19.748736 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:19.749145 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:19.749150 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:19.749154 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:19.749345 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:19.749868 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:19.749876 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:19.750249 (+ 373us) client_negotiation.cc:769] Sending connection context
0928 14:57:19.750518 (+ 269us) client_negotiation.cc:240] Negotiation successful
0928 14:57:19.751087 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":303,"threads_started":1}
I20260928 14:57:19.753847 8137 heartbeater.cc:347] Connected to a master server at 127.31.40.62:40335
I20260928 14:57:19.754204 8137 heartbeater.cc:464] Registering TS with master...
I20260928 14:57:19.755128 8137 heartbeater.cc:511] Master 127.31.40.62:40335 requested a full tablet report, sending...
I20260928 14:57:19.756868 7814 ts_manager.cc:194] Registered new tserver with Master: 9585a49fcac94e64bb952314a7ccbbe3 (127.31.40.2:38273)
I20260928 14:57:19.757712 7814 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.2@KRBTEST.COM'} at 127.31.40.2:57559
WARNING: no policy specified for kudu/127.31.40.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab.
Entry for principal kudu/127.31.40.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab.
I20260928 14:57:19.792028 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.3:0
--local_ip_for_outbound_sockets=127.31.40.3
--webserver_interface=127.31.40.3
--webserver_port=0
--tserver_master_addrs=127.31.40.62:40335
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab
--principal=kudu/127.31.40.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:35391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:57:20.136741 8143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:20.137184 8143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:20.137358 8143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:20.146062 8143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:20.146224 8143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:20.146420 8143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.3
I20260928 14:57:20.156606 8143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:35391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab
--principal=kudu/127.31.40.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.40.3
--webserver_port=0
--tserver_master_addrs=127.31.40.62:40335
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.3
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:57:20.158000 8143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:20.159368 8143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:20.173063 8148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:20.172950 8149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:20.172906 8151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:20.173759 8143 server_base.cc:1034] running on GCE node
I20260928 14:57:20.174470 8143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:20.175441 8143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:20.176688 8143 hybrid_clock.cc:648] HybridClock initialized: now 1790607440176630 us; error 48 us; skew 500 ppm
I20260928 14:57:20.177100 8143 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:57:20 dist-test-slave-j968 krb5kdc[7769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607440, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:20.180351 8143 init.cc:377] Logged in from keytab as kudu/127.31.40.3@KRBTEST.COM (short username kudu)
I20260928 14:57:20.181748 8143 webserver.cc:460] Webserver started at http://127.31.40.3:36733/ using document root <none> and password file <none>
I20260928 14:57:20.182484 8143 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:20.182588 8143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:20.182847 8143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:20.185783 8143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-2/data/instance:
uuid: "e4ed9c68567e4663ae508027ab8b6630"
format_stamp: "Formatted at 2026-09-28 14:57:20 on dist-test-slave-j968"
server_key: "085d2be879f64f999b728adb168ca63e"
server_key_iv: "fe9f57c5059e771569718e58dd5a7614"
server_key_version: "encryptionkey@0"
I20260928 14:57:20.186535 8143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-2/wal/instance:
uuid: "e4ed9c68567e4663ae508027ab8b6630"
format_stamp: "Formatted at 2026-09-28 14:57:20 on dist-test-slave-j968"
server_key: "085d2be879f64f999b728adb168ca63e"
server_key_iv: "fe9f57c5059e771569718e58dd5a7614"
server_key_version: "encryptionkey@0"
I20260928 14:57:20.192080 8143 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 14:57:20.195497 8158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:20.197073 8143 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 14:57:20.197211 8143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-2/wal
uuid: "e4ed9c68567e4663ae508027ab8b6630"
format_stamp: "Formatted at 2026-09-28 14:57:20 on dist-test-slave-j968"
server_key: "085d2be879f64f999b728adb168ca63e"
server_key_iv: "fe9f57c5059e771569718e58dd5a7614"
server_key_version: "encryptionkey@0"
I20260928 14:57:20.197373 8143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:20.217172 8143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:20.221632 8143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:20.221968 8143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:20.222973 8143 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:57:20.224431 8143 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:57:20.224507 8143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:20.224664 8143 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:57:20.224717 8143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:20.235023 8003 heartbeater.cc:503] Master 127.31.40.62:40335 was elected leader, sending a full tablet report...
I20260928 14:57:20.257187 8143 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.3:38671
I20260928 14:57:20.257225 8270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.3:38671 every 8 connection(s)
I20260928 14:57:20.258535 8143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
I20260928 14:57:20.265262 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 8143
I20260928 14:57:20.265435 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-2/wal/instance
I20260928 14:57:20.265908 31904 external_mini_cluster.cc:1428] Setting key 227701c253dc65b3b158a0f13ca68c14
Sep 28 14:57:20 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607440, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.3@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:57:20.279850 8273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:20.261260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:49709 (local address 127.31.40.62:40335)
0928 14:57:20.261761 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:20.261766 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:20.262592 (+ 826us) server_negotiation.cc:407] Connection header received
0928 14:57:20.264611 (+ 2019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:20.264617 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:20.264724 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:20.264843 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:20.266745 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:20.267731 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:20.269506 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:20.269958 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:20.270122 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:20.273618 (+ 3496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:20.273623 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:20.273626 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:20.273660 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:20.276747 (+ 3087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:20.277607 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:20.277612 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:20.277614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:20.277736 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:20.278140 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:20.278144 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:20.278146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:20.278423 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:20.278557 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:20.279233 (+ 676us) server_negotiation.cc:299] Negotiation successful
0928 14:57:20.279407 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":263,"threads_started":1}
I20260928 14:57:20.280997 8274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:20.261641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40335 (local address 127.31.40.3:49709)
0928 14:57:20.262413 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:20.262459 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:20.264319 (+ 1860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:20.265133 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:20.265142 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:20.265830 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:20.266576 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:20.266605 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:20.267906 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:20.267911 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:20.269337 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:20.269350 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:20.270216 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:20.270221 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:20.270421 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:20.271080 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:20.271106 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:20.273401 (+ 2295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:20.276894 (+ 3493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:20.276902 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:20.276924 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:20.277440 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:20.277831 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:20.277834 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:20.277837 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:20.278016 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:20.278548 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:20.278554 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:20.278871 (+ 317us) client_negotiation.cc:769] Sending connection context
0928 14:57:20.279104 (+ 233us) client_negotiation.cc:240] Negotiation successful
0928 14:57:20.279627 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":321,"threads_started":1}
I20260928 14:57:20.282431 8271 heartbeater.cc:347] Connected to a master server at 127.31.40.62:40335
I20260928 14:57:20.282761 8271 heartbeater.cc:464] Registering TS with master...
I20260928 14:57:20.283496 8271 heartbeater.cc:511] Master 127.31.40.62:40335 requested a full tablet report, sending...
I20260928 14:57:20.284891 7814 ts_manager.cc:194] Registered new tserver with Master: e4ed9c68567e4663ae508027ab8b6630 (127.31.40.3:38671)
I20260928 14:57:20.285713 7814 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.3@KRBTEST.COM'} at 127.31.40.3:49709
I20260928 14:57:20.294119 31904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.31.40.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.40.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.40.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/krb5kdc/oryx_127.31.40.1.keytab.
Entry for principal oryx/127.31.40.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/krb5kdc/oryx_127.31.40.1.keytab.
WARNING: no policy specified for oryx/127.31.40.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.40.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.40.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/krb5kdc/oryx_127.31.40.2.keytab.
Entry for principal oryx/127.31.40.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/krb5kdc/oryx_127.31.40.2.keytab.
WARNING: no policy specified for oryx/127.31.40.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.40.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.40.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/krb5kdc/oryx_127.31.40.3.keytab.
Entry for principal oryx/127.31.40.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/krb5kdc/oryx_127.31.40.3.keytab.
I20260928 14:57:20.434989 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 7875
Sep 28 14:57:20 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.1@KRBTEST.COM
I20260928 14:57:20.450726 8282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:20.436143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46756 (local address 127.31.40.1:46017)
0928 14:57:20.436804 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:20.436811 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:20.436840 (+ 29us) server_negotiation.cc:407] Connection header received
0928 14:57:20.436903 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:20.436910 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:20.437146 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:20.437285 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:20.438594 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:20.439595 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:20.440835 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:20.441423 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:20.441535 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:20.445561 (+ 4026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:20.445572 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:20.445587 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:20.445624 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:20.448345 (+ 2721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:20.448980 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:20.448987 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:20.448990 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:20.449110 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:20.449409 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:20.449413 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:20.449415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:20.449769 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:20.449992 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:20.450249 (+ 257us) server_negotiation.cc:299] Negotiation successful
0928 14:57:20.450439 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":361,"threads_started":1}
I20260928 14:57:20.452239 7998 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:57:20.537240 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 7875
I20260928 14:57:20.554850 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 8009
Sep 28 14:57:20 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.2@KRBTEST.COM
I20260928 14:57:20.572425 8284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:20.555847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38690 (local address 127.31.40.2:38273)
0928 14:57:20.556457 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:20.556464 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:20.556490 (+ 26us) server_negotiation.cc:407] Connection header received
0928 14:57:20.556549 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:20.556555 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:20.556793 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:20.556981 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:20.558277 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:20.559283 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:20.560657 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:20.561260 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:20.561406 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:20.566126 (+ 4720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:20.566139 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:20.566157 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:20.566205 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:20.569398 (+ 3193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:20.570184 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:20.570192 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:20.570194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:20.570328 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:20.570743 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:20.570747 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:20.570749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:20.571109 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:20.571435 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:20.571752 (+ 317us) server_negotiation.cc:299] Negotiation successful
0928 14:57:20.572062 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":293,"threads_started":1}
I20260928 14:57:20.573761 8132 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:57:20.657663 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 8009
I20260928 14:57:20.675576 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 8143
Sep 28 14:57:20 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.3@KRBTEST.COM
I20260928 14:57:20.693353 8286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:20.676710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34542 (local address 127.31.40.3:38671)
0928 14:57:20.677443 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:20.677449 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:20.677478 (+ 29us) server_negotiation.cc:407] Connection header received
0928 14:57:20.677538 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:20.677545 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:20.677779 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:20.678069 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:20.679395 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:20.680673 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:20.681992 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:20.682572 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:20.682715 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:20.687410 (+ 4695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:20.687424 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:20.687441 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:20.687484 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:20.690367 (+ 2883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:20.691102 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:20.691110 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:20.691112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:20.691249 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:20.691635 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:20.691641 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:20.691645 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:20.692181 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:20.692513 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:20.692828 (+ 315us) server_negotiation.cc:299] Negotiation successful
0928 14:57:20.693063 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":406,"threads_started":1}
I20260928 14:57:20.694778 8266 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:57:20.780682 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 8143
I20260928 14:57:20.799526 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 7782
I20260928 14:57:20.800858 7844 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:57:20.903908 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 7782
I20260928 14:57:20.926151 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:40335
--webserver_interface=127.31.40.62
--webserver_port=37203
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:35391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.40.62:40335
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:57:21.259939 8289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:21.260289 8289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:21.260452 8289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:21.268086 8289 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:57:21.268208 8289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:21.268292 8289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:21.268401 8289 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:57:21.268465 8289 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:57:21.278674 8289 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:35391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.40.62:40335
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:40335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.40.62
--webserver_port=37203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:57:21.280336 8289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:21.281711 8289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:21.293701 8294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:21.293673 8295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:21.294044 8297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:21.294939 8289 server_base.cc:1034] running on GCE node
I20260928 14:57:21.295812 8289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:21.297452 8289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:21.298735 8289 hybrid_clock.cc:648] HybridClock initialized: now 1790607441298661 us; error 80 us; skew 500 ppm
I20260928 14:57:21.299150 8289 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:57:21 dist-test-slave-j968 krb5kdc[7769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607441, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:21.302776 8289 init.cc:377] Logged in from keytab as kudu/127.31.40.62@KRBTEST.COM (short username kudu)
I20260928 14:57:21.304541 8289 webserver.cc:460] Webserver started at http://127.31.40.62:37203/ using document root <none> and password file <none>
I20260928 14:57:21.305325 8289 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:21.305508 8289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:21.311478 8289 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.006s
I20260928 14:57:21.314582 8304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:21.316205 8289 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 14:57:21.316417 8289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/master-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/master-0/wal
uuid: "136d295a29b14ad6a27ff1829a44c665"
format_stamp: "Formatted at 2026-09-28 14:57:18 on dist-test-slave-j968"
server_key: "9b97c69ca204bc98cd4a5329c6e4a208"
server_key_iv: "359185d4458e28c647d25565291fe42f"
server_key_version: "encryptionkey@0"
I20260928 14:57:21.317535 8289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:21.353699 8289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:21.358441 8289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:21.358749 8289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:21.378345 8289 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.62:40335
I20260928 14:57:21.378360 8355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.62:40335 every 8 connection(s)
I20260928 14:57:21.379997 8289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
I20260928 14:57:21.387053 8356 sys_catalog.cc:263] Verifying existing consensus state
I20260928 14:57:21.390034 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 8289
I20260928 14:57:21.390834 8356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 136d295a29b14ad6a27ff1829a44c665: Bootstrap starting.
I20260928 14:57:21.391052 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.1:46017
--local_ip_for_outbound_sockets=127.31.40.1
--tserver_master_addrs=127.31.40.62:40335
--webserver_port=42861
--webserver_interface=127.31.40.1
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab
--principal=kudu/127.31.40.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:35391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.31.40.1
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/krb5kdc/oryx_127.31.40.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:57:21.412597 8356 log.cc:824] T 00000000000000000000000000000000 P 136d295a29b14ad6a27ff1829a44c665: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:21.422595 8356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 136d295a29b14ad6a27ff1829a44c665: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:57:21.423161 8356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 136d295a29b14ad6a27ff1829a44c665: Bootstrap complete.
I20260928 14:57:21.432673 8356 raft_consensus.cc:348] T 00000000000000000000000000000000 P 136d295a29b14ad6a27ff1829a44c665 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "136d295a29b14ad6a27ff1829a44c665" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 40335 } }
I20260928 14:57:21.433553 8356 raft_consensus.cc:729] T 00000000000000000000000000000000 P 136d295a29b14ad6a27ff1829a44c665 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 136d295a29b14ad6a27ff1829a44c665, State: Initialized, Role: FOLLOWER
I20260928 14:57:21.434242 8356 consensus_queue.cc:260] T 00000000000000000000000000000000 P 136d295a29b14ad6a27ff1829a44c665 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "136d295a29b14ad6a27ff1829a44c665" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 40335 } }
I20260928 14:57:21.434490 8356 raft_consensus.cc:388] T 00000000000000000000000000000000 P 136d295a29b14ad6a27ff1829a44c665 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:57:21.434605 8356 raft_consensus.cc:482] T 00000000000000000000000000000000 P 136d295a29b14ad6a27ff1829a44c665 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:57:21.434811 8356 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 136d295a29b14ad6a27ff1829a44c665 [term 1 FOLLOWER]: Advancing to term 2
I20260928 14:57:21.438834 8356 raft_consensus.cc:504] T 00000000000000000000000000000000 P 136d295a29b14ad6a27ff1829a44c665 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "136d295a29b14ad6a27ff1829a44c665" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 40335 } }
I20260928 14:57:21.439376 8356 leader_election.cc:302] T 00000000000000000000000000000000 P 136d295a29b14ad6a27ff1829a44c665 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 136d295a29b14ad6a27ff1829a44c665; no voters:
I20260928 14:57:21.440063 8356 leader_election.cc:288] T 00000000000000000000000000000000 P 136d295a29b14ad6a27ff1829a44c665 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 14:57:21.440212 8360 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 136d295a29b14ad6a27ff1829a44c665 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 14:57:21.441576 8360 raft_consensus.cc:686] T 00000000000000000000000000000000 P 136d295a29b14ad6a27ff1829a44c665 [term 2 LEADER]: Becoming Leader. State: Replica: 136d295a29b14ad6a27ff1829a44c665, State: Running, Role: LEADER
I20260928 14:57:21.442263 8360 consensus_queue.cc:237] T 00000000000000000000000000000000 P 136d295a29b14ad6a27ff1829a44c665 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "136d295a29b14ad6a27ff1829a44c665" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 40335 } }
I20260928 14:57:21.442564 8356 sys_catalog.cc:564] T 00000000000000000000000000000000 P 136d295a29b14ad6a27ff1829a44c665 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:57:21.445281 8362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 136d295a29b14ad6a27ff1829a44c665 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 136d295a29b14ad6a27ff1829a44c665. Latest consensus state: current_term: 2 leader_uuid: "136d295a29b14ad6a27ff1829a44c665" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "136d295a29b14ad6a27ff1829a44c665" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 40335 } } }
I20260928 14:57:21.445320 8361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 136d295a29b14ad6a27ff1829a44c665 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "136d295a29b14ad6a27ff1829a44c665" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "136d295a29b14ad6a27ff1829a44c665" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 40335 } } }
I20260928 14:57:21.445878 8362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 136d295a29b14ad6a27ff1829a44c665 [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:21.445953 8361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 136d295a29b14ad6a27ff1829a44c665 [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:21.453872 8365 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:57:21.461215 8365 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:57:21.463311 8365 catalog_manager.cc:1259] Loaded cluster ID: 394b3c78648c4d7da553d3d717d140df
I20260928 14:57:21.463538 8365 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:57:21.469018 8365 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:57:21.471261 8365 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 136d295a29b14ad6a27ff1829a44c665: Loaded TSK: 0
I20260928 14:57:21.473409 8365 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 14:57:21.763679 8358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:21.764087 8358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:21.764269 8358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:21.772080 8358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:21.772414 8358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:21.772759 8358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.1
I20260928 14:57:21.783252 8358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:35391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/krb5kdc/oryx_127.31.40.1.keytab
--principal=oryx/127.31.40.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.1:46017
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.40.1
--webserver_port=42861
--tserver_master_addrs=127.31.40.62:40335
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.1
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:57:21.785090 8358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:21.786722 8358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:21.802232 8383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:21.802313 8385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:21.802906 8382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:21.804049 8358 server_base.cc:1034] running on GCE node
I20260928 14:57:21.804805 8358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:21.806296 8358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:21.807680 8358 hybrid_clock.cc:648] HybridClock initialized: now 1790607441807571 us; error 54 us; skew 500 ppm
I20260928 14:57:21.808516 8358 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:57:21 dist-test-slave-j968 krb5kdc[7769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607441, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.40.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:21.814009 8358 init.cc:377] Logged in from keytab as oryx/127.31.40.1@KRBTEST.COM (short username oryx)
I20260928 14:57:21.816237 8358 webserver.cc:460] Webserver started at http://127.31.40.1:42861/ using document root <none> and password file <none>
I20260928 14:57:21.817021 8358 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:21.817207 8358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:21.823400 8358 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.006s
I20260928 14:57:21.826581 8392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:21.828298 8358 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 14:57:21.828478 8358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-0/wal
uuid: "2528937497ef49e6a5cc027a5d446e46"
format_stamp: "Formatted at 2026-09-28 14:57:19 on dist-test-slave-j968"
server_key: "f62746b7954c96dcc21e4c232af96918"
server_key_iv: "5b67e09400fd698df1abab626eab5ecc"
server_key_version: "encryptionkey@0"
I20260928 14:57:21.829532 8358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:21.864058 8358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:21.868710 8358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:21.869004 8358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:21.870078 8358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:57:21.871503 8358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:57:21.871595 8358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:21.871738 8358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:57:21.871845 8358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:21.906574 8358 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.1:46017
I20260928 14:57:21.906694 8504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.1:46017 every 8 connection(s)
I20260928 14:57:21.908262 8358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
I20260928 14:57:21.918684 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 8358
I20260928 14:57:21.919374 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.2:38273
--local_ip_for_outbound_sockets=127.31.40.2
--tserver_master_addrs=127.31.40.62:40335
--webserver_port=36029
--webserver_interface=127.31.40.2
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab
--principal=kudu/127.31.40.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:35391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.31.40.2
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/krb5kdc/oryx_127.31.40.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 14:57:21 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.1@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:57:21 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.1@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:57:21.927994 8508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:21.911448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40335 (local address 127.31.40.1:44079)
0928 14:57:21.912542 (+ 1094us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:21.912601 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:21.914375 (+ 1774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:21.916026 (+ 1651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:21.916038 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:21.916685 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:21.917247 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:21.917266 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:21.918944 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:21.918950 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:21.920441 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:21.920455 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:21.921608 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:21.921616 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:21.921952 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:21.922658 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:21.922682 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:21.926921 (+ 4239us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.40.62:40335: Server oryx/127.31.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":721,"thread_start_us":524,"threads_started":1}
W20260928 14:57:21.928711 8507 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:21.911409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:44079 (local address 127.31.40.62:40335)
0928 14:57:21.912429 (+ 1020us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:21.912446 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:21.912740 (+ 294us) server_negotiation.cc:407] Connection header received
0928 14:57:21.915022 (+ 2282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:21.915036 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:21.915420 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:21.915808 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:21.917486 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:21.918744 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:21.920623 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:21.921340 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:21.921611 (+ 271us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:21.927445 (+ 5834us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.40.1:44079: BlockingRecv error: recv got EOF from 127.31.40.1:44079 (error 108)
Metrics: {"server-negotiator.queue_time_us":741,"thread_start_us":475,"threads_started":1}
W20260928 14:57:21.928855 8505 heartbeater.cc:644] Failed to heartbeat to 127.31.40.62:40335 (0 consecutive failures): Not authorized: Failed to ping master at 127.31.40.62:40335: Client connection negotiation failed: client connection to 127.31.40.62:40335: Server oryx/127.31.40.62@KRBTEST.COM not found in Kerberos database
Sep 28 14:57:21 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.1@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:57:21 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.1@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:57:21.939270 8508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:21.929507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40335 (local address 127.31.40.1:38525)
0928 14:57:21.929732 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:21.929754 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:21.929876 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:21.930212 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:21.930219 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:21.930698 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:21.930936 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:21.930951 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:21.932086 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:21.932094 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:21.933766 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:21.933784 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:21.934534 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:21.934543 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:21.934770 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:21.935406 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:21.935434 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:21.938967 (+ 3533us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.40.62:40335: Server oryx/127.31.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":63}
W20260928 14:57:21.939422 8507 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:21.929740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:38525 (local address 127.31.40.62:40335)
0928 14:57:21.929900 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:21.929903 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:21.929921 (+ 18us) server_negotiation.cc:407] Connection header received
0928 14:57:21.929994 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:21.929998 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:21.930064 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:21.930145 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:21.931131 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:21.931939 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:21.933954 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:21.934394 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:21.934488 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:21.939200 (+ 4712us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.40.1:38525: BlockingRecv error: recv got EOF from 127.31.40.1:38525 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 28 14:57:21 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.1@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:57:21 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.1@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:57:21.949719 8508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:21.940040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40335 (local address 127.31.40.1:37595)
0928 14:57:21.940263 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:21.940286 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:21.940407 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:21.940756 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:21.940764 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:21.941222 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:21.941454 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:21.941468 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:21.942623 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:21.942629 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:21.944446 (+ 1817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:21.944465 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:21.945175 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:21.945184 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:21.945420 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:21.946087 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:21.946111 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:21.949364 (+ 3253us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.40.62:40335: Server oryx/127.31.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":67}
W20260928 14:57:21.949955 8507 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:21.940185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:37595 (local address 127.31.40.62:40335)
0928 14:57:21.940322 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:21.940326 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:21.940342 (+ 16us) server_negotiation.cc:407] Connection header received
0928 14:57:21.940533 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:21.940536 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:21.940607 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:21.940683 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:21.941614 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:21.942440 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:21.944615 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:21.944994 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:21.945138 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:21.949580 (+ 4442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.40.1:37595: BlockingRecv error: recv got EOF from 127.31.40.1:37595 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260928 14:57:21.950034 8505 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 14:57:22.270069 8509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:22.270361 8509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:22.270515 8509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:22.277935 8509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:22.278060 8509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:22.278270 8509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.2
I20260928 14:57:22.288233 8509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:35391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/krb5kdc/oryx_127.31.40.2.keytab
--principal=oryx/127.31.40.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.2:38273
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.40.2
--webserver_port=36029
--tserver_master_addrs=127.31.40.62:40335
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.2
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:57:22.289630 8509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:22.291004 8509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:22.305444 8514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:22.305094 8515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:22.305509 8517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:22.306821 8509 server_base.cc:1034] running on GCE node
I20260928 14:57:22.307551 8509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:22.308681 8509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:22.309971 8509 hybrid_clock.cc:648] HybridClock initialized: now 1790607442309834 us; error 94 us; skew 500 ppm
I20260928 14:57:22.310330 8509 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:57:22 dist-test-slave-j968 krb5kdc[7769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607442, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.40.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:22.313658 8509 init.cc:377] Logged in from keytab as oryx/127.31.40.2@KRBTEST.COM (short username oryx)
I20260928 14:57:22.315277 8509 webserver.cc:460] Webserver started at http://127.31.40.2:36029/ using document root <none> and password file <none>
I20260928 14:57:22.316090 8509 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:22.316249 8509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:22.322415 8509 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 14:57:22.325695 8524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:22.327306 8509 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:57:22.327507 8509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-1/wal
uuid: "9585a49fcac94e64bb952314a7ccbbe3"
format_stamp: "Formatted at 2026-09-28 14:57:19 on dist-test-slave-j968"
server_key: "80e1ca0851efcc32e5bd35e8194f6913"
server_key_iv: "dc5aa5b1a64f93680d017522c6516101"
server_key_version: "encryptionkey@0"
I20260928 14:57:22.328621 8509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:22.341336 8509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:22.346222 8509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:22.346529 8509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:22.347582 8509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:57:22.349011 8509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:57:22.349140 8509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:22.349277 8509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:57:22.349364 8509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:22.384963 8509 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.2:38273
I20260928 14:57:22.385052 8636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.2:38273 every 8 connection(s)
I20260928 14:57:22.386858 8509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
I20260928 14:57:22.397357 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 8509
I20260928 14:57:22.398121 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.3:38671
--local_ip_for_outbound_sockets=127.31.40.3
--tserver_master_addrs=127.31.40.62:40335
--webserver_port=36733
--webserver_interface=127.31.40.3
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab
--principal=kudu/127.31.40.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:35391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.31.40.3
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/krb5kdc/oryx_127.31.40.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 14:57:22 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.2@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:57:22 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.2@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 14:57:22.406311 8507 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:22.389833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:58047 (local address 127.31.40.62:40335)
0928 14:57:22.390057 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:22.390064 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:22.391502 (+ 1438us) server_negotiation.cc:407] Connection header received
0928 14:57:22.393654 (+ 2152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:22.393658 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:22.393728 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:22.393825 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:22.395419 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:22.396400 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:22.398698 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:22.399196 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:22.399297 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:22.406116 (+ 6819us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.40.2:58047: BlockingRecv error: recv got EOF from 127.31.40.2:58047 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 14:57:22.407253 8639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:22.390403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40335 (local address 127.31.40.2:58047)
0928 14:57:22.391302 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:22.391352 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:22.393310 (+ 1958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:22.394116 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:22.394124 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:22.394678 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:22.395255 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:22.395272 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:22.396557 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:22.396561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:22.398549 (+ 1988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:22.398567 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:22.399453 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:22.399463 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:22.399795 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:22.400606 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:22.400641 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:22.405749 (+ 5108us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.40.62:40335: Server oryx/127.31.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":394,"threads_started":1}
W20260928 14:57:22.408078 8637 heartbeater.cc:644] Failed to heartbeat to 127.31.40.62:40335 (0 consecutive failures): Not authorized: Failed to ping master at 127.31.40.62:40335: Client connection negotiation failed: client connection to 127.31.40.62:40335: Server oryx/127.31.40.62@KRBTEST.COM not found in Kerberos database
Sep 28 14:57:22 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.2@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:57:22 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.2@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:57:22.420553 8639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:22.408771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40335 (local address 127.31.40.2:58571)
0928 14:57:22.409061 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:22.409099 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:22.409228 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:22.409688 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:22.409695 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:22.410234 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:22.410480 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:22.410495 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:22.411658 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:22.411667 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:22.413383 (+ 1716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:22.413410 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:22.414268 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:22.414277 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:22.414542 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:22.415327 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:22.415358 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:22.420227 (+ 4869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.40.62:40335: Server oryx/127.31.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151}
W20260928 14:57:22.420811 8507 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:22.408968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:58571 (local address 127.31.40.62:40335)
0928 14:57:22.409146 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:22.409150 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:22.409169 (+ 19us) server_negotiation.cc:407] Connection header received
0928 14:57:22.409425 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:22.409430 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:22.409507 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:22.409617 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:22.410693 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:22.411486 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:22.413601 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:22.414084 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:22.414199 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:22.420526 (+ 6327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.40.2:58571: BlockingRecv error: recv got EOF from 127.31.40.2:58571 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 28 14:57:22 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.2@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:57:22 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.2@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:57:22.430351 8639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:22.421549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40335 (local address 127.31.40.2:57713)
0928 14:57:22.421969 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:22.422000 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:22.422149 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:22.422573 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:22.422578 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:22.422916 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:22.423075 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:22.423084 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:22.424256 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:22.424260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:22.425187 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:22.425198 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:22.425937 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:22.425943 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:22.426119 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:22.426768 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:22.426786 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:22.430005 (+ 3219us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.40.62:40335: Server oryx/127.31.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 14:57:22.430455 8507 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:22.421743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:57713 (local address 127.31.40.62:40335)
0928 14:57:22.422036 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:22.422044 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:22.422066 (+ 22us) server_negotiation.cc:407] Connection header received
0928 14:57:22.422293 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:22.422298 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:22.422422 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:22.422519 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:22.423221 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:22.424109 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:22.425337 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:22.425768 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:22.425913 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:22.430236 (+ 4323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.40.2:57713: BlockingRecv error: recv got EOF from 127.31.40.2:57713 (error 108)
Metrics: {"server-negotiator.queue_time_us":152}
W20260928 14:57:22.430716 8637 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 14:57:22.765832 8640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:22.766170 8640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:22.766395 8640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:22.774731 8640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:22.774904 8640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:22.775106 8640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.3
I20260928 14:57:22.785238 8640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:35391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/krb5kdc/oryx_127.31.40.3.keytab
--principal=oryx/127.31.40.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.3:38671
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.40.3
--webserver_port=36733
--tserver_master_addrs=127.31.40.62:40335
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.3
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:57:22.786751 8640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:22.788244 8640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:22.803748 8646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:22.803915 8645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:22.804087 8648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:22.804658 8640 server_base.cc:1034] running on GCE node
I20260928 14:57:22.805474 8640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:22.806623 8640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:22.807935 8640 hybrid_clock.cc:648] HybridClock initialized: now 1790607442807797 us; error 84 us; skew 500 ppm
I20260928 14:57:22.808342 8640 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:57:22 dist-test-slave-j968 krb5kdc[7769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607442, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.40.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:22.811777 8640 init.cc:377] Logged in from keytab as oryx/127.31.40.3@KRBTEST.COM (short username oryx)
I20260928 14:57:22.813583 8640 webserver.cc:460] Webserver started at http://127.31.40.3:36733/ using document root <none> and password file <none>
I20260928 14:57:22.814338 8640 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:22.814502 8640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:22.820602 8640 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 14:57:22.824198 8655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:22.825877 8640 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 14:57:22.826134 8640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-2/wal
uuid: "e4ed9c68567e4663ae508027ab8b6630"
format_stamp: "Formatted at 2026-09-28 14:57:20 on dist-test-slave-j968"
server_key: "085d2be879f64f999b728adb168ca63e"
server_key_iv: "fe9f57c5059e771569718e58dd5a7614"
server_key_version: "encryptionkey@0"
I20260928 14:57:22.827189 8640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:22.842268 8640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:22.847297 8640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:22.847738 8640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:22.849035 8640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:57:22.850625 8640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:57:22.850756 8640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:22.850900 8640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:57:22.850996 8640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:22.888265 8640 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.3:38671
I20260928 14:57:22.888314 8768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.3:38671 every 8 connection(s)
I20260928 14:57:22.889804 8640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
I20260928 14:57:22.899055 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 8640
Sep 28 14:57:22 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.3@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:57:22 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.3@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 14:57:22.909044 8507 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:22.893076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:60737 (local address 127.31.40.62:40335)
0928 14:57:22.893263 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:22.893268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:22.894509 (+ 1241us) server_negotiation.cc:407] Connection header received
0928 14:57:22.896357 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:22.896362 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:22.896444 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:22.896546 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:22.898242 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:22.899044 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:22.901737 (+ 2693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:22.902363 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:22.902543 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:22.908696 (+ 6153us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.40.3:60737: BlockingRecv error: recv got EOF from 127.31.40.3:60737 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 14:57:22.909942 8771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:22.893580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40335 (local address 127.31.40.3:60737)
0928 14:57:22.894334 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:22.894380 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:22.896084 (+ 1704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:22.896867 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:22.896876 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:22.897464 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:22.898040 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:22.898057 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:22.899231 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:22.899236 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:22.901563 (+ 2327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:22.901578 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:22.902649 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:22.902657 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:22.902902 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:22.903725 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:22.903750 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:22.908496 (+ 4746us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.40.62:40335: Server oryx/127.31.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":283,"threads_started":1}
W20260928 14:57:22.910679 8769 heartbeater.cc:644] Failed to heartbeat to 127.31.40.62:40335 (0 consecutive failures): Not authorized: Failed to ping master at 127.31.40.62:40335: Client connection negotiation failed: client connection to 127.31.40.62:40335: Server oryx/127.31.40.62@KRBTEST.COM not found in Kerberos database
Sep 28 14:57:22 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.3@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:57:22 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.3@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:57:22.919960 8771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:22.911327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40335 (local address 127.31.40.3:54719)
0928 14:57:22.911685 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:22.911700 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:22.911848 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:22.912248 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:22.912252 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:22.912616 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:22.912763 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:22.912772 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:22.913823 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:22.913830 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:22.914739 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:22.914750 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:22.915518 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:22.915524 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:22.915704 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:22.916297 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:22.916314 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:22.919597 (+ 3283us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.40.62:40335: Server oryx/127.31.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260}
W20260928 14:57:22.920107 8507 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:22.911408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:54719 (local address 127.31.40.62:40335)
0928 14:57:22.911570 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:22.911574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:22.911779 (+ 205us) server_negotiation.cc:407] Connection header received
0928 14:57:22.912040 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:22.912046 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:22.912114 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:22.912217 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:22.912915 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:22.913674 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:22.914891 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:22.915353 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:22.915494 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:22.919814 (+ 4320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.40.3:54719: BlockingRecv error: recv got EOF from 127.31.40.3:54719 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 14:57:22.920984 8773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:22.900448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41510 (local address 127.31.40.62:40335)
0928 14:57:22.902084 (+ 1636us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:22.902089 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:22.902105 (+ 16us) server_negotiation.cc:407] Connection header received
0928 14:57:22.902148 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:22.902152 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:22.902218 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:22.902364 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:22.904452 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:22.905562 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:22.907470 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:22.908241 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:22.908394 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:22.913184 (+ 4790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:22.913197 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:22.913225 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:22.913283 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:22.916983 (+ 3700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:22.917799 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:22.917811 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:22.917821 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:22.918108 (+ 287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:22.918574 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:22.918578 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:22.918580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:22.919155 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:22.919485 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:22.920069 (+ 584us) server_negotiation.cc:299] Negotiation successful
0928 14:57:22.920605 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1525,"thread_start_us":1403,"threads_started":1}
Sep 28 14:57:22 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.3@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:57:22 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.3@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:57:22.931313 8771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:22.921172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40335 (local address 127.31.40.3:52755)
0928 14:57:22.921376 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:22.921397 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:22.921517 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:22.922069 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:22.922074 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:22.922480 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:22.922635 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:22.922644 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:22.924587 (+ 1943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:22.924593 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:22.925892 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:22.925906 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:22.926666 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:22.926673 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:22.926873 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:22.927511 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:22.927531 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:22.930975 (+ 3444us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.40.62:40335: Server oryx/127.31.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 14:57:22.931787 8773 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:22.921471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:52755 (local address 127.31.40.62:40335)
0928 14:57:22.921698 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:22.921705 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:22.921720 (+ 15us) server_negotiation.cc:407] Connection header received
0928 14:57:22.921801 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:22.921804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:22.921875 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:22.922016 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:22.922799 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:22.923982 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:22.926046 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:22.926511 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:22.926653 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:22.931322 (+ 4669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.40.3:52755: BlockingRecv error: recv got EOF from 127.31.40.3:52755 (error 108)
Metrics: {"server-negotiator.queue_time_us":128}
W20260928 14:57:22.931761 8769 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 14:57:22 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.1@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:57:22 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.1@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:57:22.959764 8774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:22.950739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40335 (local address 127.31.40.1:52769)
0928 14:57:22.951336 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:22.951356 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:22.951516 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:22.951835 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:22.951838 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:22.952362 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:22.952586 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:22.952602 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:22.953688 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:22.953691 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:22.954641 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:22.954652 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:22.955445 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:22.955449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:22.955599 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:22.956328 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:22.956372 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:22.959508 (+ 3136us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.40.62:40335: Server oryx/127.31.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":322,"threads_started":1}
W20260928 14:57:22.960071 8773 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:22.950851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:52769 (local address 127.31.40.62:40335)
0928 14:57:22.951258 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:22.951264 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:22.951463 (+ 199us) server_negotiation.cc:407] Connection header received
0928 14:57:22.951627 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:22.951631 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:22.951702 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:22.951785 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:22.952745 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:22.953559 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:22.954802 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:22.955279 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:22.955392 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:22.959736 (+ 4344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.40.1:52769: BlockingRecv error: recv got EOF from 127.31.40.1:52769 (error 108)
Metrics: {"server-negotiator.queue_time_us":125}
Sep 28 14:57:23 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.2@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:57:23 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.2@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:57:23.440707 8775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:23.431480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40335 (local address 127.31.40.2:36241)
0928 14:57:23.432193 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:23.432216 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:23.432401 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:23.432857 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:23.432862 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:23.433428 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:23.433780 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:23.433791 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:23.434884 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:23.434889 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:23.435833 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:23.435844 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:23.436624 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:23.436630 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:23.436807 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:23.437402 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:23.437420 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:23.440465 (+ 3045us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.40.62:40335: Server oryx/127.31.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":415,"threads_started":1}
W20260928 14:57:23.441002 8773 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:23.431604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:36241 (local address 127.31.40.62:40335)
0928 14:57:23.431841 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:23.431851 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:23.432314 (+ 463us) server_negotiation.cc:407] Connection header received
0928 14:57:23.432554 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:23.432560 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:23.432668 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:23.432772 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:23.433926 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:23.434747 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:23.436011 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:23.436432 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:23.436565 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:23.440735 (+ 4170us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.40.2:36241: BlockingRecv error: recv got EOF from 127.31.40.2:36241 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 28 14:57:23 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.3@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:57:23 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.3@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:57:23.945705 8776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:23.932522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40335 (local address 127.31.40.3:44617)
0928 14:57:23.933132 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:23.933152 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:23.933270 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:23.933694 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:23.933699 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:23.934340 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:23.934572 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:23.934588 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:23.935899 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:23.935907 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:23.936964 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:23.936977 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:23.937718 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:23.937725 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:23.937910 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:23.942376 (+ 4466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:23.942402 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:23.945474 (+ 3072us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.40.62:40335: Server oryx/127.31.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":350,"threads_started":1}
W20260928 14:57:23.945972 8773 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:23.932629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:44617 (local address 127.31.40.62:40335)
0928 14:57:23.932915 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:23.932923 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:23.933238 (+ 315us) server_negotiation.cc:407] Connection header received
0928 14:57:23.933427 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:23.933434 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:23.933525 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:23.933641 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:23.934730 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:23.935735 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:23.937113 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:23.937539 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:23.937683 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:23.945708 (+ 8025us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.40.3:44617: BlockingRecv error: recv got EOF from 127.31.40.3:44617 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 28 14:57:23 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.1@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:57:23 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.1@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:57:23.969805 8777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:23.960818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40335 (local address 127.31.40.1:56777)
0928 14:57:23.961582 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:23.961599 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:23.961766 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:23.962148 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:23.962153 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:23.962625 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:23.962769 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:23.962778 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:23.963794 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:23.963800 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:23.964878 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:23.964892 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:23.965592 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:23.965598 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:23.965771 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:23.966514 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:23.966536 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:23.969566 (+ 3030us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.40.62:40335: Server oryx/127.31.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":652,"thread_start_us":530,"threads_started":1}
W20260928 14:57:23.970018 8773 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:23.960894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:56777 (local address 127.31.40.62:40335)
0928 14:57:23.961156 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:23.961165 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:23.961690 (+ 525us) server_negotiation.cc:407] Connection header received
0928 14:57:23.961940 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:23.961944 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:23.962007 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:23.962117 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:23.962926 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:23.963649 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:23.965038 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:23.965437 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:23.965575 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:23.969757 (+ 4182us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.40.1:56777: BlockingRecv error: recv got EOF from 127.31.40.1:56777 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 28 14:57:24 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.2@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:57:24 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.2@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:57:24.451464 8778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:24.441670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40335 (local address 127.31.40.2:56337)
0928 14:57:24.442307 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:24.442324 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:24.442538 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:24.442923 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:24.442927 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:24.443427 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:24.443578 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:24.443587 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:24.444872 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:24.444878 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:24.445869 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:24.445883 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:24.446676 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:24.446682 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:24.446870 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:24.447582 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:24.447602 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:24.451196 (+ 3594us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.40.62:40335: Server oryx/127.31.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":381,"threads_started":1}
W20260928 14:57:24.451712 8773 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:24.441806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:56337 (local address 127.31.40.62:40335)
0928 14:57:24.442167 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:24.442175 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:24.442447 (+ 272us) server_negotiation.cc:407] Connection header received
0928 14:57:24.442686 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:24.442692 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:24.442774 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:24.442899 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:24.443739 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:24.444708 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:24.446047 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:24.446505 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:24.446659 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:24.451447 (+ 4788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.40.2:56337: BlockingRecv error: recv got EOF from 127.31.40.2:56337 (error 108)
Metrics: {"server-negotiator.queue_time_us":201}
Sep 28 14:57:24 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.3@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:57:24 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.3@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:57:24.955217 8779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:24.946661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40335 (local address 127.31.40.3:58991)
0928 14:57:24.947214 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:24.947235 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:24.947391 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:24.947761 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:24.947766 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:24.948220 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:24.948365 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:24.948374 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:24.949416 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:24.949423 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:24.950334 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:24.950345 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:24.951084 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:24.951090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:24.951252 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:24.951917 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:24.951947 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:24.954953 (+ 3006us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.40.62:40335: Server oryx/127.31.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":329,"threads_started":1}
W20260928 14:57:24.955317 8773 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:24.946854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:58991 (local address 127.31.40.62:40335)
0928 14:57:24.947084 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:24.947090 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:24.947316 (+ 226us) server_negotiation.cc:407] Connection header received
0928 14:57:24.947516 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:24.947522 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:24.947609 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:24.947710 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:24.948539 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:24.949285 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:24.950498 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:24.950918 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:24.951057 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:24.955139 (+ 4082us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.40.3:58991: BlockingRecv error: recv got EOF from 127.31.40.3:58991 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 28 14:57:24 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.1@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:57:24 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.1@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:57:24.979854 8780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:24.970863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40335 (local address 127.31.40.1:42855)
0928 14:57:24.971399 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:24.971414 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:24.971539 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:24.971849 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:24.971852 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:24.972288 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:24.972493 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:24.972502 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:24.973803 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:24.973807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:24.974739 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:24.974751 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:24.975443 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:24.975447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:24.975601 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:24.976292 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:24.976313 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:24.979493 (+ 3180us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.40.62:40335: Server oryx/127.31.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":291,"threads_started":1}
W20260928 14:57:24.980026 8773 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:24.970874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:42855 (local address 127.31.40.62:40335)
0928 14:57:24.971102 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:24.971106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:24.971502 (+ 396us) server_negotiation.cc:407] Connection header received
0928 14:57:24.971647 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:24.971651 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:24.971716 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:24.971801 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:24.972639 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:24.973654 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:24.974896 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:24.975315 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:24.975425 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:24.979724 (+ 4299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.40.1:42855: BlockingRecv error: recv got EOF from 127.31.40.1:42855 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Sep 28 14:57:25 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.2@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:57:25 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.2@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:57:25.461635 8781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:25.452475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40335 (local address 127.31.40.2:41919)
0928 14:57:25.453082 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:25.453098 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:25.453219 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:25.453643 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:25.453646 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:25.454075 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:25.454248 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:25.454258 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:25.455509 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:25.455512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:25.456588 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:25.456601 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:25.457316 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:25.457320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:25.457462 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:25.458075 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:25.458097 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:25.461254 (+ 3157us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.40.62:40335: Server oryx/127.31.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":327,"threads_started":1}
W20260928 14:57:25.461683 8773 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:25.452595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:41919 (local address 127.31.40.62:40335)
0928 14:57:25.452777 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:25.452781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:25.453186 (+ 405us) server_negotiation.cc:407] Connection header received
0928 14:57:25.453404 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:25.453409 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:25.453484 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:25.453583 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:25.454434 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:25.455347 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:25.456715 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:25.457169 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:25.457270 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:25.461460 (+ 4190us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.40.2:41919: BlockingRecv error: recv got EOF from 127.31.40.2:41919 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 28 14:57:25 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.3@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:57:25 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.3@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:57:25.966074 8782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:25.956468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40335 (local address 127.31.40.3:44451)
0928 14:57:25.957204 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:25.957234 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:25.957374 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:25.957804 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:25.957807 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:25.958233 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:25.958436 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:25.958445 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:25.959680 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:25.959685 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:25.960730 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:25.960742 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:25.961448 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:25.961453 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:25.961585 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:25.962224 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:25.962252 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:25.965635 (+ 3383us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.40.62:40335: Server oryx/127.31.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":460,"threads_started":1}
W20260928 14:57:25.966243 8773 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:25.956525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:44451 (local address 127.31.40.62:40335)
0928 14:57:25.956695 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:25.956731 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:25.957353 (+ 622us) server_negotiation.cc:407] Connection header received
0928 14:57:25.957545 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:25.957549 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:25.957654 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:25.957743 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:25.958609 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:25.959525 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:25.960861 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:25.961289 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:25.961395 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:25.965889 (+ 4494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.40.3:44451: BlockingRecv error: recv got EOF from 127.31.40.3:44451 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 28 14:57:25 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.1@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:57:25 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.1@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:57:25.989449 8783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:25.980936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40335 (local address 127.31.40.1:47043)
0928 14:57:25.981502 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:25.981519 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:25.981631 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:25.982018 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:25.982021 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:25.982367 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:25.982512 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:25.982520 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:25.983527 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:25.983531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:25.984648 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:25.984663 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:25.985357 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:25.985361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:25.985523 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:25.986174 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:25.986191 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:25.989101 (+ 2910us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.40.62:40335: Server oryx/127.31.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":298,"threads_started":1}
W20260928 14:57:25.989555 8773 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:25.981134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:47043 (local address 127.31.40.62:40335)
0928 14:57:25.981427 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:25.981431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:25.981593 (+ 162us) server_negotiation.cc:407] Connection header received
0928 14:57:25.981734 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:25.981738 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:25.981799 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:25.981908 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:25.982640 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:25.983372 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:25.984797 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:25.985216 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:25.985318 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:25.989311 (+ 3993us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.40.1:47043: BlockingRecv error: recv got EOF from 127.31.40.1:47043 (error 108)
Metrics: {"server-negotiator.queue_time_us":176}
Sep 28 14:57:26 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.2@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:57:26 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.2@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:57:26.471661 8784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:26.462775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40335 (local address 127.31.40.2:47873)
0928 14:57:26.463295 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:26.463313 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:26.463456 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:26.463895 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:26.463900 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:26.464339 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:26.464582 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:26.464597 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:26.465904 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:26.465908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:26.466926 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:26.466939 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:26.467671 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:26.467676 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:26.467809 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:26.468535 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:26.468552 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:26.471380 (+ 2828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.40.62:40335: Server oryx/127.31.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":281,"threads_started":1}
W20260928 14:57:26.471834 8773 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:26.462984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:47873 (local address 127.31.40.62:40335)
0928 14:57:26.463225 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:26.463234 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:26.463425 (+ 191us) server_negotiation.cc:407] Connection header received
0928 14:57:26.463586 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:26.463592 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:26.463676 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:26.463790 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:26.464747 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:26.465756 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:26.467105 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:26.467531 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:26.467634 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:26.471591 (+ 3957us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.40.2:47873: BlockingRecv error: recv got EOF from 127.31.40.2:47873 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Sep 28 14:57:26 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.3@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:57:26 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.3@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:57:26.976189 8786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:26.967326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40335 (local address 127.31.40.3:49905)
0928 14:57:26.967857 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:26.967902 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:26.968013 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:26.968365 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:26.968369 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:26.968748 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:26.968897 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:26.968905 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:26.970114 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:26.970118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:26.971119 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:26.971132 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:26.971974 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:26.971980 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:26.972135 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:26.972813 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:26.972832 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:26.975953 (+ 3121us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.40.62:40335: Server oryx/127.31.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":336,"threads_started":1}
W20260928 14:57:26.976430 8773 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:26.967522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:49905 (local address 127.31.40.62:40335)
0928 14:57:26.967912 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:26.967920 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:26.967974 (+ 54us) server_negotiation.cc:407] Connection header received
0928 14:57:26.968138 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:26.968142 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:26.968228 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:26.968321 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:26.969059 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:26.969962 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:26.971287 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:26.971740 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:26.971849 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:26.976164 (+ 4315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.40.3:49905: BlockingRecv error: recv got EOF from 127.31.40.3:49905 (error 108)
Metrics: {"server-negotiator.queue_time_us":184}
Sep 28 14:57:26 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.1@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:57:26 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.1@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:57:26.999934 8787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:26.990459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40335 (local address 127.31.40.1:34255)
0928 14:57:26.991102 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:26.991119 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:26.991234 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:26.991592 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:26.991596 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:26.992181 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:26.992354 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:26.992366 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:26.993551 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:26.993557 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:26.994667 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:26.994682 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:26.995452 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:26.995457 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:26.995595 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:26.996260 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:26.996280 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:26.999552 (+ 3272us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.40.62:40335: Server oryx/127.31.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":289,"threads_started":1}
W20260928 14:57:27.000128 8773 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:26.990564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:34255 (local address 127.31.40.62:40335)
0928 14:57:26.990741 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:26.990745 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:26.991225 (+ 480us) server_negotiation.cc:407] Connection header received
0928 14:57:26.991359 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:26.991363 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:26.991427 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:26.991522 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:26.992569 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:26.993382 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:26.994834 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:26.995283 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:26.995428 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:26.999772 (+ 4344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.40.1:34255: BlockingRecv error: recv got EOF from 127.31.40.1:34255 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 28 14:57:27 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.2@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:57:27 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.2@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:57:27.482100 8788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:27.472818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40335 (local address 127.31.40.2:38041)
0928 14:57:27.473557 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:27.473580 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:27.473731 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:27.474136 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:27.474141 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:27.474621 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:27.474795 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:27.474804 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:27.476006 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:27.476010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:27.477003 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:27.477015 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:27.477751 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:27.477756 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:27.477923 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:27.478634 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:27.478658 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:27.481713 (+ 3055us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.40.62:40335: Server oryx/127.31.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":394,"threads_started":1}
W20260928 14:57:27.482141 8773 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:27.472923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:38041 (local address 127.31.40.62:40335)
0928 14:57:27.473130 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:27.473136 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:27.473697 (+ 561us) server_negotiation.cc:407] Connection header received
0928 14:57:27.473878 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:27.473883 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:27.473964 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:27.474107 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:27.475014 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:27.475775 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:27.477139 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:27.477605 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:27.477703 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:27.481929 (+ 4226us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.40.2:38041: BlockingRecv error: recv got EOF from 127.31.40.2:38041 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 28 14:57:27 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.3@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:57:27 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.3@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:57:27.986220 8789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:27.977544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40335 (local address 127.31.40.3:56481)
0928 14:57:27.978163 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:27.978185 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:27.978308 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:27.978691 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:27.978696 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:27.979090 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:27.979244 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:27.979253 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:27.980306 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:27.980310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:27.981283 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:27.981296 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:27.982073 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:27.982078 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:27.982218 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:27.982978 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:27.982999 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:27.985960 (+ 2961us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.40.62:40335: Server oryx/127.31.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":395,"threads_started":1}
W20260928 14:57:27.986387 8773 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:27.977693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:56481 (local address 127.31.40.62:40335)
0928 14:57:27.977867 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:27.977876 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:27.978274 (+ 398us) server_negotiation.cc:407] Connection header received
0928 14:57:27.978424 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:27.978430 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:27.978527 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:27.978610 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:27.979379 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:27.980181 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:27.981459 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:27.981906 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:27.982023 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:27.986147 (+ 4124us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.40.3:56481: BlockingRecv error: recv got EOF from 127.31.40.3:56481 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 28 14:57:28 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.1@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:57:28 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.1@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:57:28.010246 8790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:28.001299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40335 (local address 127.31.40.1:35123)
0928 14:57:28.001905 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:28.001921 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:28.002082 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:28.002447 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:28.002452 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:28.002859 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:28.003024 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:28.003033 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:28.004135 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:28.004141 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:28.005039 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:28.005050 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:28.005772 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:28.005778 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:28.005967 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:28.006635 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:28.006652 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:28.009997 (+ 3345us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.40.62:40335: Server oryx/127.31.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":352,"threads_started":1}
W20260928 14:57:28.010483 8773 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:28.001315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:35123 (local address 127.31.40.62:40335)
0928 14:57:28.001519 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:28.001524 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:28.002010 (+ 486us) server_negotiation.cc:407] Connection header received
0928 14:57:28.002232 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:28.002237 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:28.002311 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:28.002419 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:28.003164 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:28.003998 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:28.005194 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:28.005605 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:28.005746 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:28.010214 (+ 4468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.40.1:35123: BlockingRecv error: recv got EOF from 127.31.40.1:35123 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 28 14:57:28 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.2@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:57:28 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.2@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:57:28.493224 8791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:28.483290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40335 (local address 127.31.40.2:56401)
0928 14:57:28.483915 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:28.483936 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:28.484249 (+ 313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:28.484736 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:28.484740 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:28.485121 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:28.485277 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:28.485286 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:28.486756 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:28.486762 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:28.487948 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:28.487963 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:28.488771 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:28.488777 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:28.488976 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:28.489645 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:28.489664 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:28.492890 (+ 3226us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.40.62:40335: Server oryx/127.31.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":362,"threads_started":1}
W20260928 14:57:28.493472 8773 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:28.483433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:56401 (local address 127.31.40.62:40335)
0928 14:57:28.483713 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:28.483721 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:28.484080 (+ 359us) server_negotiation.cc:407] Connection header received
0928 14:57:28.484458 (+ 378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:28.484463 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:28.484533 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:28.484665 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:28.485430 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:28.486601 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:28.488111 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:28.488585 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:28.488745 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:28.493195 (+ 4450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.40.2:56401: BlockingRecv error: recv got EOF from 127.31.40.2:56401 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Sep 28 14:57:28 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.3@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:57:28 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.3@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:57:29.002741 8792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:28.987405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40335 (local address 127.31.40.3:33653)
0928 14:57:28.988059 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:28.988082 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:28.988246 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:28.988649 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:28.988653 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:28.989299 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:28.989452 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:28.989460 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:28.990979 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:28.990983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:28.991970 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:28.991983 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:28.992772 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:28.992778 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:28.992988 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:28.998995 (+ 6007us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:28.999016 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:29.002396 (+ 3380us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.40.62:40335: Server oryx/127.31.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":303,"threads_started":1}
W20260928 14:57:29.002924 8773 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:28.987599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:33653 (local address 127.31.40.62:40335)
0928 14:57:28.987861 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:28.987899 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:28.988177 (+ 278us) server_negotiation.cc:407] Connection header received
0928 14:57:28.988382 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:28.988388 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:28.988471 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:28.988595 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:28.989643 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:28.990822 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:28.992122 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:28.992599 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:28.992721 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:29.002673 (+ 9952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.40.3:33653: BlockingRecv error: recv got EOF from 127.31.40.3:33653 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
Sep 28 14:57:29 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.1@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:57:29 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.1@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:57:29.020939 8793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:29.011328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40335 (local address 127.31.40.1:51465)
0928 14:57:29.012159 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:29.012176 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:29.012345 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:29.012745 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:29.012750 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:29.013184 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:29.013420 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:29.013433 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:29.014769 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:29.014773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:29.016063 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:29.016078 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:29.016751 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:29.016756 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:29.016933 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:29.017598 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:29.017616 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:29.020694 (+ 3078us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.40.62:40335: Server oryx/127.31.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":734,"thread_start_us":318,"threads_started":1}
W20260928 14:57:29.021435 8773 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:29.011570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:51465 (local address 127.31.40.62:40335)
0928 14:57:29.011844 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:29.011850 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:29.012269 (+ 419us) server_negotiation.cc:407] Connection header received
0928 14:57:29.012520 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:29.012525 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:29.012588 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:29.012703 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:29.013598 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:29.014617 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:29.016190 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:29.016599 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:29.016697 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:29.020971 (+ 4274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.40.1:51465: BlockingRecv error: recv got EOF from 127.31.40.1:51465 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 28 14:57:29 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.2@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:57:29 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.2@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:57:29.503360 8794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:29.494406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40335 (local address 127.31.40.2:46369)
0928 14:57:29.495088 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:29.495110 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:29.495273 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:29.495670 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:29.495675 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:29.496219 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:29.496385 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:29.496395 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:29.497477 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:29.497482 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:29.498407 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:29.498417 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:29.499162 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:29.499168 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:29.499345 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:29.500041 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:29.500060 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:29.503123 (+ 3063us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.40.62:40335: Server oryx/127.31.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":407,"threads_started":1}
W20260928 14:57:29.503706 8773 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:29.494638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:46369 (local address 127.31.40.62:40335)
0928 14:57:29.494883 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:29.494891 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:29.495191 (+ 300us) server_negotiation.cc:407] Connection header received
0928 14:57:29.495424 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:29.495432 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:29.495517 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:29.495634 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:29.496587 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:29.497319 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:29.498566 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:29.498996 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:29.499131 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:29.503487 (+ 4356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.40.2:46369: BlockingRecv error: recv got EOF from 127.31.40.2:46369 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Sep 28 14:57:30 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.3@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:57:30 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.3@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:57:30.013240 8795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:30.003972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40335 (local address 127.31.40.3:56755)
0928 14:57:30.004679 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:30.004708 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:30.004855 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:30.005405 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:30.005409 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:30.005830 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:30.006027 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:30.006037 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:30.007083 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:30.007088 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:30.008172 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:30.008186 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:30.009018 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:30.009022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:30.009143 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:30.009843 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:30.009863 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:30.012953 (+ 3090us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.40.62:40335: Server oryx/127.31.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":346,"threads_started":1}
W20260928 14:57:30.013675 8796 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:30.004481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:56755 (local address 127.31.40.62:40335)
0928 14:57:30.005025 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:30.005032 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:30.005056 (+ 24us) server_negotiation.cc:407] Connection header received
0928 14:57:30.005137 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:30.005142 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:30.005232 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:30.005322 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:30.006164 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:30.006927 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:30.008341 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:30.008845 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:30.008980 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:30.013202 (+ 4222us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.40.3:56755: BlockingRecv error: recv got EOF from 127.31.40.3:56755 (error 108)
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":317,"threads_started":1}
Sep 28 14:57:30 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.1@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:57:30 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.1@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:57:30.031216 8797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:30.022013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40335 (local address 127.31.40.1:43629)
0928 14:57:30.022643 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:30.022659 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:30.022813 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:30.023212 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:30.023216 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:30.023657 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:30.023801 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:30.023811 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:30.024905 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:30.024912 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:30.025875 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:30.025888 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:30.026779 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:30.026788 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:30.026942 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:30.027609 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:30.027627 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:30.030981 (+ 3354us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.40.62:40335: Server oryx/127.31.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":319,"threads_started":1}
W20260928 14:57:30.031409 8796 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:30.022323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:43629 (local address 127.31.40.62:40335)
0928 14:57:30.022527 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:30.022534 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:30.022743 (+ 209us) server_negotiation.cc:407] Connection header received
0928 14:57:30.022965 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:30.022969 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:30.023076 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:30.023192 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:30.023999 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:30.024738 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:30.026011 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:30.026426 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:30.026537 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:30.031174 (+ 4637us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.40.1:43629: BlockingRecv error: recv got EOF from 127.31.40.1:43629 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 28 14:57:30 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.2@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:57:30 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.2@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:57:30.514249 8798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:30.504601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40335 (local address 127.31.40.2:58579)
0928 14:57:30.505237 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:30.505254 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:30.505384 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:30.505752 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:30.505756 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:30.506151 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:30.506297 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:30.506307 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:30.507511 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:30.507515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:30.508626 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:30.508641 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:30.509501 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:30.509507 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:30.509645 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:30.510490 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:30.510511 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:30.513883 (+ 3372us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.40.62:40335: Server oryx/127.31.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":358,"threads_started":1}
W20260928 14:57:30.514398 8796 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:30.504704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:58579 (local address 127.31.40.62:40335)
0928 14:57:30.504907 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:30.504912 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:30.505364 (+ 452us) server_negotiation.cc:407] Connection header received
0928 14:57:30.505524 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:30.505528 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:30.505595 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:30.505684 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:30.506443 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:30.507370 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:30.508783 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:30.509289 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:30.509390 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:30.514098 (+ 4708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.40.2:58579: BlockingRecv error: recv got EOF from 127.31.40.2:58579 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 28 14:57:31 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.3@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:57:31 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.3@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:57:31.023423 8799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:31.014343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40335 (local address 127.31.40.3:40991)
0928 14:57:31.014943 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:31.014965 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:31.015090 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:31.015445 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:31.015449 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:31.015849 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:31.016085 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:31.016094 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:31.017247 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:31.017251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:31.018182 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:31.018194 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:31.018951 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:31.018958 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:31.019135 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:31.019861 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:31.019920 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:31.023175 (+ 3255us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.40.62:40335: Server oryx/127.31.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":438,"threads_started":1}
W20260928 14:57:31.023736 8796 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:31.014484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:40991 (local address 127.31.40.62:40335)
0928 14:57:31.014688 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:31.014696 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:31.015071 (+ 375us) server_negotiation.cc:407] Connection header received
0928 14:57:31.015197 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:31.015201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:31.015280 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:31.015366 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:31.016236 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:31.017097 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:31.018316 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:31.018739 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:31.018891 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:31.023476 (+ 4585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.40.3:40991: BlockingRecv error: recv got EOF from 127.31.40.3:40991 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 28 14:57:31 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.1@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:57:31 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.1@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:57:31.041937 8800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:31.032335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40335 (local address 127.31.40.1:56243)
0928 14:57:31.033159 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:31.033178 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:31.033341 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:31.033696 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:31.033703 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:31.034098 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:31.034327 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:31.034337 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:31.035371 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:31.035377 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:31.036412 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:31.036432 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:31.037156 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:31.037162 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:31.037348 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:31.037976 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:31.037994 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:31.041553 (+ 3559us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.40.62:40335: Server oryx/127.31.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":709,"thread_start_us":579,"threads_started":1}
W20260928 14:57:31.042078 8796 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:31.032419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:56243 (local address 127.31.40.62:40335)
0928 14:57:31.032640 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:31.032646 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:31.033270 (+ 624us) server_negotiation.cc:407] Connection header received
0928 14:57:31.033461 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:31.033467 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:31.033537 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:31.033653 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:31.034475 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:31.035223 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:31.036568 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:31.037001 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:31.037144 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:31.041784 (+ 4640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.40.1:56243: BlockingRecv error: recv got EOF from 127.31.40.1:56243 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 28 14:57:31 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.2@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:57:31 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.2@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:57:31.524808 8801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:31.515608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40335 (local address 127.31.40.2:42021)
0928 14:57:31.516188 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:31.516209 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:31.516359 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:31.516749 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:31.516754 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:31.517202 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:31.517376 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:31.517385 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:31.518593 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:31.518598 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:31.519540 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:31.519551 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:31.520440 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:31.520446 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:31.520650 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:31.521260 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:31.521279 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:31.524540 (+ 3261us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.40.62:40335: Server oryx/127.31.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":287,"threads_started":1}
W20260928 14:57:31.525059 8796 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:31.515710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:42021 (local address 127.31.40.62:40335)
0928 14:57:31.516011 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:31.516017 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:31.516389 (+ 372us) server_negotiation.cc:407] Connection header received
0928 14:57:31.516498 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:31.516502 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:31.516593 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:31.516712 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:31.517536 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:31.518443 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:31.519704 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:31.520250 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:31.520391 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:31.524773 (+ 4382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.40.2:42021: BlockingRecv error: recv got EOF from 127.31.40.2:42021 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 28 14:57:32 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.3@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:57:32 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.3@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:57:32.034284 8802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:32.024543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40335 (local address 127.31.40.3:39413)
0928 14:57:32.025099 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:32.025119 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:32.025292 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:32.025743 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:32.025749 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:32.026407 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:32.026666 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:32.026682 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:32.028054 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:32.028059 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:32.029029 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:32.029042 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:32.029725 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:32.029730 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:32.029852 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:32.030612 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:32.030631 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:32.034032 (+ 3401us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.40.62:40335: Server oryx/127.31.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":342,"threads_started":1}
W20260928 14:57:32.034579 8796 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:32.024607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:39413 (local address 127.31.40.62:40335)
0928 14:57:32.024847 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:32.024856 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:32.025215 (+ 359us) server_negotiation.cc:407] Connection header received
0928 14:57:32.025434 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:32.025438 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:32.025533 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:32.025638 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:32.026864 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:32.027792 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:32.029173 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:32.029584 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:32.029684 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:32.034250 (+ 4566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.40.3:39413: BlockingRecv error: recv got EOF from 127.31.40.3:39413 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 28 14:57:32 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.1@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:57:32 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.1@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:57:32.051765 8803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:32.042977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40335 (local address 127.31.40.1:35057)
0928 14:57:32.043566 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:32.043588 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:32.043707 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:32.044119 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:32.044124 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:32.044502 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:32.044648 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:32.044658 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:32.045699 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:32.045704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:32.046656 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:32.046669 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:32.047407 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:32.047414 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:32.047589 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:32.048266 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:32.048284 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:32.051407 (+ 3123us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.40.62:40335: Server oryx/127.31.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":308,"threads_started":1}
W20260928 14:57:32.051941 8796 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:32.043144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:35057 (local address 127.31.40.62:40335)
0928 14:57:32.043330 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:32.043335 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:32.043696 (+ 361us) server_negotiation.cc:407] Connection header received
0928 14:57:32.043847 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:32.043853 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:32.043951 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:32.044060 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:32.044803 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:32.045548 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:32.046807 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:32.047242 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:32.047383 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:32.051672 (+ 4289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.40.1:35057: BlockingRecv error: recv got EOF from 127.31.40.1:35057 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 28 14:57:32 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.2@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:57:32 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.2@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:57:32.535797 8804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:32.525789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40335 (local address 127.31.40.2:53551)
0928 14:57:32.526289 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:32.526307 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:32.526434 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:32.526758 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:32.526761 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:32.527194 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:32.527355 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:32.527365 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:32.528725 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:32.528729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:32.529690 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:32.529703 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:32.530628 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:32.530633 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:32.530825 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:32.531609 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:32.531628 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:32.535447 (+ 3819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.40.62:40335: Server oryx/127.31.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":286,"threads_started":1}
W20260928 14:57:32.536124 8796 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:32.525881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:53551 (local address 127.31.40.62:40335)
0928 14:57:32.526146 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:32.526154 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:32.526393 (+ 239us) server_negotiation.cc:407] Connection header received
0928 14:57:32.526553 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:32.526557 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:32.526627 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:32.526716 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:32.527538 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:32.528568 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:32.529847 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:32.530421 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:32.530622 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:32.535777 (+ 5155us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.40.2:53551: BlockingRecv error: recv got EOF from 127.31.40.2:53551 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 28 14:57:33 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.3@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:57:33 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.3@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:57:33.044963 8805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:33.035436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40335 (local address 127.31.40.3:33311)
0928 14:57:33.036036 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:33.036057 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:33.036214 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:33.037152 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:33.037157 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:33.037574 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:33.037728 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:33.037737 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:33.038764 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:33.038769 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:33.039701 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:33.039713 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:33.040525 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:33.040530 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:33.040711 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:33.041317 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:33.041334 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:33.044617 (+ 3283us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.40.62:40335: Server oryx/127.31.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":311,"threads_started":1}
W20260928 14:57:33.045117 8796 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:33.035657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:33311 (local address 127.31.40.62:40335)
0928 14:57:33.035898 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:33.035903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:33.036147 (+ 244us) server_negotiation.cc:407] Connection header received
0928 14:57:33.036913 (+ 766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:33.036917 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:33.036992 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:33.037113 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:33.037882 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:33.038616 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:33.039895 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:33.040350 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:33.040498 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:33.044865 (+ 4367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.40.3:33311: BlockingRecv error: recv got EOF from 127.31.40.3:33311 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 28 14:57:33 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.1@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:57:33 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.1@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:57:33.062716 8806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:33.052979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40335 (local address 127.31.40.1:52443)
0928 14:57:33.053560 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:33.053578 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:33.053694 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:33.054081 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:33.054085 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:33.054516 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:33.054703 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:33.054713 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:33.055814 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:33.055819 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:33.056958 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:33.056974 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:33.057779 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:33.057784 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:33.057918 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:33.058687 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:33.058707 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:33.062432 (+ 3725us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.40.62:40335: Server oryx/127.31.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":342,"threads_started":1}
W20260928 14:57:33.062894 8796 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:33.053082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:52443 (local address 127.31.40.62:40335)
0928 14:57:33.053268 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:33.053273 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:33.053656 (+ 383us) server_negotiation.cc:407] Connection header received
0928 14:57:33.053801 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:33.053805 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:33.053865 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:33.053991 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:33.054940 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:33.055676 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:33.057116 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:33.057587 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:33.057694 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:33.062638 (+ 4944us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.40.1:52443: BlockingRecv error: recv got EOF from 127.31.40.1:52443 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 28 14:57:33 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.2@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:57:33 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.2@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:57:33.546730 8807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:33.536967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40335 (local address 127.31.40.2:58119)
0928 14:57:33.537466 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:33.537483 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:33.537598 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:33.537872 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:33.537876 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:33.538490 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:33.538700 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:33.538710 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:33.539753 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:33.539758 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:33.541005 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:33.541017 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:33.541812 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:33.541817 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:33.542000 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:33.542799 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:33.542818 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:33.546343 (+ 3525us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.40.62:40335: Server oryx/127.31.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":259,"threads_started":1}
W20260928 14:57:33.546834 8796 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:33.537034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:58119 (local address 127.31.40.62:40335)
0928 14:57:33.537279 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:33.537285 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:33.537567 (+ 282us) server_negotiation.cc:407] Connection header received
0928 14:57:33.537684 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:33.537687 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:33.537749 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:33.537828 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:33.538861 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:33.539620 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:33.541152 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:33.541642 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:33.541745 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:33.546579 (+ 4834us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.40.2:58119: BlockingRecv error: recv got EOF from 127.31.40.2:58119 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 28 14:57:34 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.3@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:57:34 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.3@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:57:34.061947 8808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:34.046133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40335 (local address 127.31.40.3:37289)
0928 14:57:34.046799 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:34.046831 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:34.046949 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:34.047516 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:34.047521 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:34.048125 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:34.048393 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:34.048406 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:34.049622 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:34.049626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:34.050566 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:34.050579 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:34.051313 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:34.051317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:34.051476 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:34.057908 (+ 6432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:34.057935 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:34.061669 (+ 3734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.40.62:40335: Server oryx/127.31.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":426,"threads_started":1}
W20260928 14:57:34.062388 8796 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:34.046255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:37289 (local address 127.31.40.62:40335)
0928 14:57:34.046498 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:34.046505 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:34.046896 (+ 391us) server_negotiation.cc:407] Connection header received
0928 14:57:34.047084 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:34.047089 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:34.047182 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:34.047296 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:34.048556 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:34.049479 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:34.050723 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:34.051173 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:34.051279 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:34.062033 (+ 10754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.40.3:37289: BlockingRecv error: recv got EOF from 127.31.40.3:37289 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 28 14:57:34 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.1@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:57:34 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.1@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:57:34.073933 8809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:34.063674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40335 (local address 127.31.40.1:53497)
0928 14:57:34.064323 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:34.064340 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:34.064510 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:34.064961 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:34.064966 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:34.065503 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:34.065686 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:34.065696 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:34.066838 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:34.066843 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:34.068336 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:34.068352 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:34.069245 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:34.069249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:34.069390 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:34.070153 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:34.070172 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:34.073582 (+ 3410us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.40.62:40335: Server oryx/127.31.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":397,"threads_started":1}
W20260928 14:57:34.073992 8796 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:34.063745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:53497 (local address 127.31.40.62:40335)
0928 14:57:34.063978 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:34.063985 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:34.064492 (+ 507us) server_negotiation.cc:407] Connection header received
0928 14:57:34.064710 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:34.064715 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:34.064791 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:34.064904 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:34.065849 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:34.066683 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:34.068522 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:34.069057 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:34.069186 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:34.073774 (+ 4588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.40.1:53497: BlockingRecv error: recv got EOF from 127.31.40.1:53497 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 28 14:57:34 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.2@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:57:34 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.2@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:57:34.556779 8810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:34.547938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40335 (local address 127.31.40.2:38477)
0928 14:57:34.548502 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:34.548519 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:34.548666 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:34.549003 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:34.549006 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:34.549393 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:34.549551 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:34.549559 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:34.550621 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:34.550625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:34.551655 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:34.551669 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:34.552482 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:34.552488 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:34.552682 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:34.553378 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:34.553397 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:34.556466 (+ 3069us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.40.62:40335: Server oryx/127.31.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":288,"threads_started":1}
W20260928 14:57:34.557011 8796 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:34.548201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:38477 (local address 127.31.40.62:40335)
0928 14:57:34.548448 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:34.548465 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:34.548602 (+ 137us) server_negotiation.cc:407] Connection header received
0928 14:57:34.548793 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:34.548797 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:34.548863 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:34.548950 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:34.549679 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:34.550450 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:34.551803 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:34.552300 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:34.552463 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:34.556688 (+ 4225us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.40.2:38477: BlockingRecv error: recv got EOF from 127.31.40.2:38477 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
Sep 28 14:57:35 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.3@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:57:35 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.3@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:57:35.073027 8811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:35.063064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40335 (local address 127.31.40.3:47651)
0928 14:57:35.064285 (+ 1221us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:35.064312 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:35.064507 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:35.064996 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:35.065001 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:35.065441 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:35.065602 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:35.065610 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:35.066674 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:35.066679 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:35.067620 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:35.067631 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:35.068420 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:35.068426 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:35.068624 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:35.069335 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:35.069354 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:35.072654 (+ 3300us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.40.62:40335: Server oryx/127.31.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":918,"spinlock_wait_cycles":273280,"thread_start_us":735,"threads_started":1}
W20260928 14:57:35.073215 8812 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:35.063167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:47651 (local address 127.31.40.62:40335)
0928 14:57:35.064005 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:35.064012 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:35.064431 (+ 419us) server_negotiation.cc:407] Connection header received
0928 14:57:35.064669 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:35.064674 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:35.064748 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:35.064863 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:35.065753 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:35.066518 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:35.067767 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:35.068265 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:35.068414 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:35.072918 (+ 4504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.40.3:47651: BlockingRecv error: recv got EOF from 127.31.40.3:47651 (error 108)
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":380,"threads_started":1}
Sep 28 14:57:35 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.1@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:57:35 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.1@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:57:35.084051 8813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:35.075022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40335 (local address 127.31.40.1:35797)
0928 14:57:35.075963 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:35.075981 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:35.076085 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:35.076499 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:35.076505 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:35.076896 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:35.077050 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:35.077058 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:35.078142 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:35.078146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:35.079064 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:35.079076 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:35.079796 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:35.079800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:35.080001 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:35.080628 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:35.080649 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:35.083649 (+ 3000us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.40.62:40335: Server oryx/127.31.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":806,"thread_start_us":655,"threads_started":1}
W20260928 14:57:35.084090 8812 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:35.075065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:35797 (local address 127.31.40.62:40335)
0928 14:57:35.075231 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:35.075235 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:35.076046 (+ 811us) server_negotiation.cc:407] Connection header received
0928 14:57:35.076199 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:35.076204 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:35.076287 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:35.076418 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:35.077180 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:35.077996 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:35.079211 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:35.079627 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:35.079723 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:35.083840 (+ 4117us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.40.1:35797: BlockingRecv error: recv got EOF from 127.31.40.1:35797 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 28 14:57:35 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.2@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:57:35 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.2@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:57:35.566622 8814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:35.557911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40335 (local address 127.31.40.2:54795)
0928 14:57:35.558531 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:35.558557 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:35.558697 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:35.558994 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:35.558997 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:35.559365 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:35.559522 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:35.559529 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:35.560757 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:35.560762 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:35.561772 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:35.561786 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:35.562644 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:35.562649 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:35.562771 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:35.563549 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:35.563569 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:35.566370 (+ 2801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.40.62:40335: Server oryx/127.31.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":350,"threads_started":1}
W20260928 14:57:35.567075 8812 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:35.558031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:54795 (local address 127.31.40.62:40335)
0928 14:57:35.558194 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:35.558198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:35.558656 (+ 458us) server_negotiation.cc:407] Connection header received
0928 14:57:35.558793 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:35.558796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:35.558869 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:35.558951 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:35.559723 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:35.560591 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:35.561970 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:35.562460 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:35.562573 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:35.566623 (+ 4050us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.40.2:54795: BlockingRecv error: recv got EOF from 127.31.40.2:54795 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 28 14:57:36 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.3@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:57:36 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.3@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:57:36.083737 8815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:36.074240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40335 (local address 127.31.40.3:33549)
0928 14:57:36.074856 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:36.074877 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:36.075006 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:36.075451 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:36.075455 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:36.075860 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:36.076145 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:36.076154 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:36.077353 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:36.077358 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:36.078308 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:36.078319 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:36.079257 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:36.079264 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:36.079449 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:36.080216 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:36.080244 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:36.083360 (+ 3116us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.40.62:40335: Server oryx/127.31.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":316,"threads_started":1}
W20260928 14:57:36.083891 8816 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:36.074307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:33549 (local address 127.31.40.62:40335)
0928 14:57:36.074867 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:36.074872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:36.074960 (+ 88us) server_negotiation.cc:407] Connection header received
0928 14:57:36.075164 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:36.075169 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:36.075250 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:36.075353 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:36.076300 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:36.077212 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:36.078490 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:36.079031 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:36.079217 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:36.083591 (+ 4374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.40.3:33549: BlockingRecv error: recv got EOF from 127.31.40.3:33549 (error 108)
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":324,"threads_started":1}
Sep 28 14:57:36 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.1@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:57:36 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.1@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:57:36.093403 8817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:36.085127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40335 (local address 127.31.40.1:52777)
0928 14:57:36.085635 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:36.085650 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:36.085799 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:36.086138 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:36.086143 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:36.086500 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:36.086665 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:36.086674 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:36.087646 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:36.087652 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:36.088656 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:36.088669 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:36.089384 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:36.089390 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:36.089551 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:36.090216 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:36.090234 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:36.093038 (+ 2804us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.40.62:40335: Server oryx/127.31.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":286,"threads_started":1}
W20260928 14:57:36.093466 8816 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:36.085215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:52777 (local address 127.31.40.62:40335)
0928 14:57:36.085516 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:36.085522 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:36.085728 (+ 206us) server_negotiation.cc:407] Connection header received
0928 14:57:36.085935 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:36.085940 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:36.086008 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:36.086106 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:36.086800 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:36.087506 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:36.088810 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:36.089228 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:36.089363 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:36.093247 (+ 3884us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.40.1:52777: BlockingRecv error: recv got EOF from 127.31.40.1:52777 (error 108)
Metrics: {"server-negotiator.queue_time_us":148}
Sep 28 14:57:36 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.2@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:57:36 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.2@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:57:36.576992 8818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:36.567761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40335 (local address 127.31.40.2:59467)
0928 14:57:36.568446 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:36.568463 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:36.568584 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:36.568953 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:36.568957 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:36.569315 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:36.569528 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:36.569538 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:36.570747 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:36.570751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:36.571983 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:36.572003 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:36.572953 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:36.572960 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:36.573084 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:36.573680 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:36.573698 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:36.576654 (+ 2956us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.40.62:40335: Server oryx/127.31.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":405,"threads_started":1}
W20260928 14:57:36.577343 8816 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:36.567996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:59467 (local address 127.31.40.62:40335)
0928 14:57:36.568215 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:36.568220 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:36.568544 (+ 324us) server_negotiation.cc:407] Connection header received
0928 14:57:36.568702 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:36.568706 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:36.568771 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:36.568868 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:36.569712 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:36.570589 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:36.572163 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:36.572795 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:36.572920 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:36.576863 (+ 3943us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.40.2:59467: BlockingRecv error: recv got EOF from 127.31.40.2:59467 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Sep 28 14:57:37 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.3@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:57:37 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.3@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:57:37.096709 8819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:37.084862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40335 (local address 127.31.40.3:54299)
0928 14:57:37.085308 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:37.085329 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:37.085448 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:37.088838 (+ 3390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:37.088843 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:37.089239 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:37.089427 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:37.089436 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:37.090644 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:37.090648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:37.091539 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:37.091551 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:37.092362 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:37.092368 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:37.092537 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:37.093211 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:37.093229 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:37.096318 (+ 3089us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.40.62:40335: Server oryx/127.31.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":231,"threads_started":1}
W20260928 14:57:37.096815 8820 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:37.084930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:54299 (local address 127.31.40.62:40335)
0928 14:57:37.088457 (+ 3527us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:37.088464 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:37.088486 (+ 22us) server_negotiation.cc:407] Connection header received
0928 14:57:37.088544 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:37.088550 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:37.088625 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:37.088767 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:37.089585 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:37.090482 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:37.091700 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:37.092199 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:37.092331 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:37.096566 (+ 4235us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.40.3:54299: BlockingRecv error: recv got EOF from 127.31.40.3:54299 (error 108)
Metrics: {"server-negotiator.queue_time_us":3331,"thread_start_us":3181,"threads_started":1}
Sep 28 14:57:37 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.1@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:57:37 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.1@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:57:37.104275 8821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:37.094663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40335 (local address 127.31.40.1:58069)
0928 14:57:37.095570 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:37.095592 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:37.095713 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:37.096259 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:37.096267 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:37.096749 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:37.097024 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:37.097038 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:37.098083 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:37.098089 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:37.099004 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:37.099016 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:37.099714 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:37.099720 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:37.099946 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:37.100597 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:37.100620 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:37.103999 (+ 3379us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.40.62:40335: Server oryx/127.31.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":756,"thread_start_us":636,"threads_started":1}
W20260928 14:57:37.104812 8822 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:37.094721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:58069 (local address 127.31.40.62:40335)
0928 14:57:37.095959 (+ 1238us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:37.095966 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:37.095987 (+ 21us) server_negotiation.cc:407] Connection header received
0928 14:57:37.096043 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:37.096046 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:37.096099 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:37.096185 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:37.097178 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:37.097942 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:37.099154 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:37.099570 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:37.099703 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:37.104521 (+ 4818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.40.1:58069: BlockingRecv error: recv got EOF from 127.31.40.1:58069 (error 108)
Metrics: {"server-negotiator.queue_time_us":1098,"thread_start_us":968,"threads_started":1}
Sep 28 14:57:37 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.2@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:57:37 dist-test-slave-j968 krb5kdc[7769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.40.2@KRBTEST.COM for oryx/127.31.40.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:57:37.587963 8823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:37.578416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40335 (local address 127.31.40.2:45861)
0928 14:57:37.579002 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:37.579019 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:37.579191 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:37.579582 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:37.579590 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:37.580059 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:37.580249 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:37.580257 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:37.581320 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:37.581327 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:37.582256 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:37.582268 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:37.582969 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:37.582975 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:37.583148 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:37.584080 (+ 932us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:37.584098 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:37.587532 (+ 3434us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.40.62:40335: Server oryx/127.31.40.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":290,"threads_started":1}
W20260928 14:57:37.587996 8822 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:37.578502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:45861 (local address 127.31.40.62:40335)
0928 14:57:37.578785 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:37.578791 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:37.579122 (+ 331us) server_negotiation.cc:407] Connection header received
0928 14:57:37.579333 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:37.579338 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:37.579429 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:37.579539 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:37.580401 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:37.581167 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:37.582401 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:37.582819 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:37.582957 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:37.587751 (+ 4794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.40.2:45861: BlockingRecv error: recv got EOF from 127.31.40.2:45861 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
I20260928 14:57:37.900481 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 8358
I20260928 14:57:37.914531 8825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:37.901742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41364 (local address 127.31.40.1:46017)
0928 14:57:37.902373 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:37.902380 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:37.902409 (+ 29us) server_negotiation.cc:407] Connection header received
0928 14:57:37.902472 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:37.902479 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:37.902706 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:37.902874 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:37.904433 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:37.905563 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:37.906819 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:37.907415 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:37.907537 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:37.911136 (+ 3599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:37.911145 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:37.911161 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:37.911198 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:37.914028 (+ 2830us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.31.40.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 14:57:37.914303 (+ 275us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41364: Request ticket server kudu/127.31.40.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":352,"threads_started":1}
W20260928 14:57:37.914665 8825 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41364: Request ticket server kudu/127.31.40.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260928 14:57:37.914911 31904 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 8358: Not authorized: Client connection negotiation failed: client connection to 127.31.40.1:46017: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.31.40.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 14:57:37.915161 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 8358
I20260928 14:57:37.933624 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 8509
I20260928 14:57:37.947150 8826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:37.934961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57814 (local address 127.31.40.2:38273)
0928 14:57:37.935636 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:37.935642 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:37.935673 (+ 31us) server_negotiation.cc:407] Connection header received
0928 14:57:37.935735 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:37.935741 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:37.936053 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:37.936211 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:37.937651 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:37.938811 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:37.940226 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:37.940820 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:37.940950 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:37.943758 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:37.943772 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:37.943792 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:37.943847 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:37.946615 (+ 2768us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.31.40.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 14:57:37.946899 (+ 284us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57814: Request ticket server kudu/127.31.40.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":338,"threads_started":1}
W20260928 14:57:37.947237 8826 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57814: Request ticket server kudu/127.31.40.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260928 14:57:37.947501 31904 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 8509: Not authorized: Client connection negotiation failed: client connection to 127.31.40.2:38273: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.31.40.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 14:57:37.947634 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 8509
I20260928 14:57:37.965205 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 8640
I20260928 14:57:37.978032 8827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:37.966329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33166 (local address 127.31.40.3:38671)
0928 14:57:37.966951 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:37.966959 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:37.966988 (+ 29us) server_negotiation.cc:407] Connection header received
0928 14:57:37.967053 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:37.967060 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:37.967295 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:37.967473 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:37.969006 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:37.970113 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:37.971372 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:37.972091 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:37.972197 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:37.974758 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:37.974766 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:37.974781 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:37.974828 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:37.977526 (+ 2698us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.31.40.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 14:57:37.977785 (+ 259us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33166: Request ticket server kudu/127.31.40.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":327,"threads_started":1}
W20260928 14:57:37.978173 8827 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33166: Request ticket server kudu/127.31.40.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260928 14:57:37.978322 31904 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 8640: Not authorized: Client connection negotiation failed: client connection to 127.31.40.3:38671: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.31.40.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 14:57:37.978513 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 8640
I20260928 14:57:37.995548 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 8289
I20260928 14:57:37.997099 8351 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:57:38.113962 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 8289
2026-09-28T14:57:38Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21365 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T14:57:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:57:38Z Disabled control of system clock
I20260928 14:57:38.200215 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:40541
--webserver_interface=127.31.40.62
--webserver_port=0
--builtin_ntp_servers=127.31.40.20:33891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.40.62:40541
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 14:57:38.545610 8832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:38.545933 8832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:38.546025 8832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:38.553345 8832 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:57:38.553465 8832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:38.553503 8832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:38.553552 8832 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:57:38.553596 8832 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:57:38.562875 8832 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:33891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.40.62:40541
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:40541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:57:38.564235 8832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:38.566007 8832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:38.579097 8840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:38.579308 8837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:38.579097 8838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:38.580098 8832 server_base.cc:1034] running on GCE node
I20260928 14:57:38.580896 8832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:38.582434 8832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:38.583640 8832 hybrid_clock.cc:648] HybridClock initialized: now 1790607458583599 us; error 33 us; skew 500 ppm
I20260928 14:57:38.584081 8832 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 14:57:38.586443 8832 webserver.cc:460] Webserver started at http://127.31.40.62:38815/ using document root <none> and password file <none>
I20260928 14:57:38.587183 8832 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:38.587298 8832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:38.587659 8832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:38.590651 8832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/master-0/data/instance:
uuid: "e133d7cd20f04e4abcc83c822e7b4cca"
format_stamp: "Formatted at 2026-09-28 14:57:38 on dist-test-slave-j968"
server_key: "965ec99e501c947c343e9bf84ae9bd6e"
server_key_iv: "019f8b661e801210dc87b623118e6bb9"
server_key_version: "encryptionkey@0"
I20260928 14:57:38.591465 8832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/master-0/wal/instance:
uuid: "e133d7cd20f04e4abcc83c822e7b4cca"
format_stamp: "Formatted at 2026-09-28 14:57:38 on dist-test-slave-j968"
server_key: "965ec99e501c947c343e9bf84ae9bd6e"
server_key_iv: "019f8b661e801210dc87b623118e6bb9"
server_key_version: "encryptionkey@0"
I20260928 14:57:38.597047 8832 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260928 14:57:38.601282 8846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:38.602865 8832 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 14:57:38.603068 8832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/master-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/master-0/wal
uuid: "e133d7cd20f04e4abcc83c822e7b4cca"
format_stamp: "Formatted at 2026-09-28 14:57:38 on dist-test-slave-j968"
server_key: "965ec99e501c947c343e9bf84ae9bd6e"
server_key_iv: "019f8b661e801210dc87b623118e6bb9"
server_key_version: "encryptionkey@0"
I20260928 14:57:38.603313 8832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:38.627033 8832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:38.628091 8832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:38.628425 8832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:38.649525 8832 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.62:40541
I20260928 14:57:38.649551 8897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.62:40541 every 8 connection(s)
I20260928 14:57:38.651049 8832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
I20260928 14:57:38.654811 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 8832
I20260928 14:57:38.655062 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/master-0/wal/instance
I20260928 14:57:38.655718 8898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:38.655794 31904 external_mini_cluster.cc:1428] Setting key bc74e3b47a36be561e14b1d260c39744
I20260928 14:57:38.670674 8898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e133d7cd20f04e4abcc83c822e7b4cca: Bootstrap starting.
I20260928 14:57:38.671528 8901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:38.658191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51076 (local address 127.31.40.62:40541)
0928 14:57:38.659435 (+ 1244us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:38.659446 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:38.659490 (+ 44us) server_negotiation.cc:407] Connection header received
0928 14:57:38.661220 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:38.661246 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:38.661776 (+ 530us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:38.662188 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:38.662897 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:38.664480 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:38.666316 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:38.667098 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:38.667250 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:38.668854 (+ 1604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:38.668862 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:38.668879 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:57:38.669488 (+ 609us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:38.669573 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:57:38.669600 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:38.669689 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:38.669956 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 14:57:38.670351 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":765,"thread_start_us":507,"threads_started":1}
I20260928 14:57:38.674603 8898 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e133d7cd20f04e4abcc83c822e7b4cca: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:38.675684 8898 log.cc:824] T 00000000000000000000000000000000 P e133d7cd20f04e4abcc83c822e7b4cca: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:38.678668 8898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e133d7cd20f04e4abcc83c822e7b4cca: No bootstrap required, opened a new log
I20260928 14:57:38.686730 8898 raft_consensus.cc:348] T 00000000000000000000000000000000 P e133d7cd20f04e4abcc83c822e7b4cca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e133d7cd20f04e4abcc83c822e7b4cca" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 40541 } }
I20260928 14:57:38.687207 8898 raft_consensus.cc:374] T 00000000000000000000000000000000 P e133d7cd20f04e4abcc83c822e7b4cca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:38.687330 8898 raft_consensus.cc:729] T 00000000000000000000000000000000 P e133d7cd20f04e4abcc83c822e7b4cca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e133d7cd20f04e4abcc83c822e7b4cca, State: Initialized, Role: FOLLOWER
I20260928 14:57:38.688032 8898 consensus_queue.cc:260] T 00000000000000000000000000000000 P e133d7cd20f04e4abcc83c822e7b4cca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e133d7cd20f04e4abcc83c822e7b4cca" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 40541 } }
I20260928 14:57:38.688284 8898 raft_consensus.cc:388] T 00000000000000000000000000000000 P e133d7cd20f04e4abcc83c822e7b4cca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:57:38.688432 8898 raft_consensus.cc:482] T 00000000000000000000000000000000 P e133d7cd20f04e4abcc83c822e7b4cca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:57:38.688611 8898 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e133d7cd20f04e4abcc83c822e7b4cca [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:38.691098 8898 raft_consensus.cc:504] T 00000000000000000000000000000000 P e133d7cd20f04e4abcc83c822e7b4cca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e133d7cd20f04e4abcc83c822e7b4cca" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 40541 } }
I20260928 14:57:38.691653 8898 leader_election.cc:302] T 00000000000000000000000000000000 P e133d7cd20f04e4abcc83c822e7b4cca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e133d7cd20f04e4abcc83c822e7b4cca; no voters:
I20260928 14:57:38.692373 8898 leader_election.cc:288] T 00000000000000000000000000000000 P e133d7cd20f04e4abcc83c822e7b4cca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:57:38.692546 8903 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e133d7cd20f04e4abcc83c822e7b4cca [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:57:38.693318 8903 raft_consensus.cc:686] T 00000000000000000000000000000000 P e133d7cd20f04e4abcc83c822e7b4cca [term 1 LEADER]: Becoming Leader. State: Replica: e133d7cd20f04e4abcc83c822e7b4cca, State: Running, Role: LEADER
I20260928 14:57:38.693919 8903 consensus_queue.cc:237] T 00000000000000000000000000000000 P e133d7cd20f04e4abcc83c822e7b4cca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e133d7cd20f04e4abcc83c822e7b4cca" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 40541 } }
I20260928 14:57:38.694713 8898 sys_catalog.cc:564] T 00000000000000000000000000000000 P e133d7cd20f04e4abcc83c822e7b4cca [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:57:38.696584 8904 sys_catalog.cc:455] T 00000000000000000000000000000000 P e133d7cd20f04e4abcc83c822e7b4cca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e133d7cd20f04e4abcc83c822e7b4cca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e133d7cd20f04e4abcc83c822e7b4cca" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 40541 } } }
I20260928 14:57:38.696952 8904 sys_catalog.cc:458] T 00000000000000000000000000000000 P e133d7cd20f04e4abcc83c822e7b4cca [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:38.697821 8908 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:57:38.697672 8905 sys_catalog.cc:455] T 00000000000000000000000000000000 P e133d7cd20f04e4abcc83c822e7b4cca [sys.catalog]: SysCatalogTable state changed. Reason: New leader e133d7cd20f04e4abcc83c822e7b4cca. Latest consensus state: current_term: 1 leader_uuid: "e133d7cd20f04e4abcc83c822e7b4cca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e133d7cd20f04e4abcc83c822e7b4cca" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 40541 } } }
I20260928 14:57:38.698884 8905 sys_catalog.cc:458] T 00000000000000000000000000000000 P e133d7cd20f04e4abcc83c822e7b4cca [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:38.705776 8908 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:57:38.715327 8908 catalog_manager.cc:1347] Generated new cluster ID: a44bd523c4f34aaaa301686e432df90e
I20260928 14:57:38.715466 8908 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:57:38.732168 8908 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:57:38.734220 8908 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:57:38.753979 8908 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e133d7cd20f04e4abcc83c822e7b4cca: Generated new TSK 0
I20260928 14:57:38.755012 8908 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:57:38.760664 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.1:0
--local_ip_for_outbound_sockets=127.31.40.1
--webserver_interface=127.31.40.1
--webserver_port=0
--tserver_master_addrs=127.31.40.62:40541
--builtin_ntp_servers=127.31.40.20:33891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 14:57:39.119279 8923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:39.119637 8923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:39.119798 8923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:39.127369 8923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:39.127485 8923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:39.127589 8923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.1
I20260928 14:57:39.137441 8923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:33891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.40.1
--webserver_port=0
--tserver_master_addrs=127.31.40.62:40541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.1
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:57:39.138796 8923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:39.140799 8923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:39.156373 8931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:39.156538 8928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:39.156921 8929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:39.157141 8923 server_base.cc:1034] running on GCE node
I20260928 14:57:39.157892 8923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:39.158921 8923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:39.160156 8923 hybrid_clock.cc:648] HybridClock initialized: now 1790607459160092 us; error 58 us; skew 500 ppm
I20260928 14:57:39.160544 8923 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 14:57:39.166582 8923 webserver.cc:460] Webserver started at http://127.31.40.1:42861/ using document root <none> and password file <none>
I20260928 14:57:39.167390 8923 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:39.167544 8923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:39.167934 8923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:39.170917 8923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-0/data/instance:
uuid: "30e266dffd4647d6ba8c2a354bc896de"
format_stamp: "Formatted at 2026-09-28 14:57:39 on dist-test-slave-j968"
server_key: "02bd3db5e9c17ef94a837dc6127f8d27"
server_key_iv: "ef0241d9a8f56110a1a52ff3fd8d73d4"
server_key_version: "encryptionkey@0"
I20260928 14:57:39.171757 8923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-0/wal/instance:
uuid: "30e266dffd4647d6ba8c2a354bc896de"
format_stamp: "Formatted at 2026-09-28 14:57:39 on dist-test-slave-j968"
server_key: "02bd3db5e9c17ef94a837dc6127f8d27"
server_key_iv: "ef0241d9a8f56110a1a52ff3fd8d73d4"
server_key_version: "encryptionkey@0"
I20260928 14:57:39.177459 8923 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 14:57:39.181082 8937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:39.182569 8923 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 14:57:39.182757 8923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-0/wal
uuid: "30e266dffd4647d6ba8c2a354bc896de"
format_stamp: "Formatted at 2026-09-28 14:57:39 on dist-test-slave-j968"
server_key: "02bd3db5e9c17ef94a837dc6127f8d27"
server_key_iv: "ef0241d9a8f56110a1a52ff3fd8d73d4"
server_key_version: "encryptionkey@0"
I20260928 14:57:39.183032 8923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:39.197166 8923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:39.198132 8923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:39.198441 8923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:39.199631 8923 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:57:39.201444 8923 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:57:39.201563 8923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:39.201699 8923 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:57:39.201783 8923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:39.238529 8923 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.1:40359
I20260928 14:57:39.238590 9049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.1:40359 every 8 connection(s)
I20260928 14:57:39.239923 8923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
I20260928 14:57:39.240509 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 8923
I20260928 14:57:39.240715 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-0/wal/instance
I20260928 14:57:39.241555 31904 external_mini_cluster.cc:1428] Setting key 2897179fc3eb54d360a957ec3855a70d
I20260928 14:57:39.245015 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.2:0
--local_ip_for_outbound_sockets=127.31.40.2
--webserver_interface=127.31.40.2
--webserver_port=0
--tserver_master_addrs=127.31.40.62:40541
--builtin_ntp_servers=127.31.40.20:33891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 14:57:39.253556 9052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:39.243017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:56079 (local address 127.31.40.62:40541)
0928 14:57:39.243472 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:39.243477 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:39.245036 (+ 1559us) server_negotiation.cc:407] Connection header received
0928 14:57:39.246937 (+ 1901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:39.246942 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:39.247035 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:39.247152 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:39.248334 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:39.249062 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:39.250605 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:39.251029 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:39.251169 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:39.252645 (+ 1476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:39.252650 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:39.252653 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:57:39.252671 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:39.252721 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:57:39.252733 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:39.252849 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:39.253082 (+ 233us) server_negotiation.cc:299] Negotiation successful
0928 14:57:39.253203 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":241,"threads_started":1}
I20260928 14:57:39.254660 9053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:39.243782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40541 (local address 127.31.40.1:56079)
0928 14:57:39.244790 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:39.244834 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:39.246656 (+ 1822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:39.247392 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:39.247407 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:39.247505 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:39.248164 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:39.248182 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:39.249226 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:39.249230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:39.250457 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:39.250470 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:39.251259 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:39.251265 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:39.251501 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:39.252250 (+ 749us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:57:39.252252 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:39.252298 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:57:39.252376 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:39.252831 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:39.252843 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:39.252851 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 14:57:39.253063 (+ 212us) client_negotiation.cc:240] Negotiation successful
0928 14:57:39.253482 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":307,"threads_started":1}
I20260928 14:57:39.256286 9050 heartbeater.cc:347] Connected to a master server at 127.31.40.62:40541
I20260928 14:57:39.256620 9050 heartbeater.cc:464] Registering TS with master...
I20260928 14:57:39.257565 9050 heartbeater.cc:511] Master 127.31.40.62:40541 requested a full tablet report, sending...
I20260928 14:57:39.259799 8863 ts_manager.cc:194] Registered new tserver with Master: 30e266dffd4647d6ba8c2a354bc896de (127.31.40.1:40359)
I20260928 14:57:39.261411 8863 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.40.1:56079
W20260928 14:57:39.597857 9054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:39.598258 9054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:39.598404 9054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:39.606063 9054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:39.606184 9054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:39.606348 9054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.2
I20260928 14:57:39.615803 9054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:33891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.40.2
--webserver_port=0
--tserver_master_addrs=127.31.40.62:40541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.2
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:57:39.617295 9054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:39.619212 9054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:39.633811 9062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:39.633821 9059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:39.634179 9060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:39.635072 9054 server_base.cc:1034] running on GCE node
I20260928 14:57:39.635831 9054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:39.637068 9054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:39.638302 9054 hybrid_clock.cc:648] HybridClock initialized: now 1790607459638242 us; error 58 us; skew 500 ppm
I20260928 14:57:39.638679 9054 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 14:57:39.641376 9054 webserver.cc:460] Webserver started at http://127.31.40.2:44355/ using document root <none> and password file <none>
I20260928 14:57:39.642163 9054 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:39.642282 9054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:39.642640 9054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:39.645663 9054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-1/data/instance:
uuid: "711207c18ccb424bb223ad958e45c938"
format_stamp: "Formatted at 2026-09-28 14:57:39 on dist-test-slave-j968"
server_key: "3ae5430f10dac96cb21e1077f191c7b0"
server_key_iv: "6dc99216e84893963daad46ec602bc25"
server_key_version: "encryptionkey@0"
I20260928 14:57:39.646492 9054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-1/wal/instance:
uuid: "711207c18ccb424bb223ad958e45c938"
format_stamp: "Formatted at 2026-09-28 14:57:39 on dist-test-slave-j968"
server_key: "3ae5430f10dac96cb21e1077f191c7b0"
server_key_iv: "6dc99216e84893963daad46ec602bc25"
server_key_version: "encryptionkey@0"
I20260928 14:57:39.652132 9054 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 14:57:39.655802 9068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:39.657231 9054 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:57:39.657425 9054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-1/wal
uuid: "711207c18ccb424bb223ad958e45c938"
format_stamp: "Formatted at 2026-09-28 14:57:39 on dist-test-slave-j968"
server_key: "3ae5430f10dac96cb21e1077f191c7b0"
server_key_iv: "6dc99216e84893963daad46ec602bc25"
server_key_version: "encryptionkey@0"
I20260928 14:57:39.657691 9054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:39.675562 9054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:39.676532 9054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:39.676826 9054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:39.677855 9054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:57:39.679404 9054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:57:39.679527 9054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:39.679669 9054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:57:39.679761 9054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:39.713650 9054 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.2:39583
I20260928 14:57:39.713742 9180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.2:39583 every 8 connection(s)
I20260928 14:57:39.715032 9054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
I20260928 14:57:39.721385 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 9054
I20260928 14:57:39.721580 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-1/wal/instance
I20260928 14:57:39.722102 31904 external_mini_cluster.cc:1428] Setting key 10cf69253af0e34698343a5ddbbbed9a
I20260928 14:57:39.725270 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.3:0
--local_ip_for_outbound_sockets=127.31.40.3
--webserver_interface=127.31.40.3
--webserver_port=0
--tserver_master_addrs=127.31.40.62:40541
--builtin_ntp_servers=127.31.40.20:33891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 14:57:39.729990 9052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:39.717903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:53441 (local address 127.31.40.62:40541)
0928 14:57:39.718083 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:39.718088 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:39.719669 (+ 1581us) server_negotiation.cc:407] Connection header received
0928 14:57:39.721501 (+ 1832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:39.721506 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:39.721601 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:39.721715 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:39.722965 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:39.724181 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:39.726895 (+ 2714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:39.727352 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:39.727476 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:39.729208 (+ 1732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:39.729213 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:39.729217 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:57:39.729236 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:39.729268 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:57:39.729281 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:39.729356 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:39.729666 (+ 310us) server_negotiation.cc:299] Negotiation successful
0928 14:57:39.729755 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 14:57:39.731575 9183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:39.718497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40541 (local address 127.31.40.2:53441)
0928 14:57:39.719469 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:39.719541 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:39.721217 (+ 1676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:39.721973 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:39.721981 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:39.722072 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:39.722717 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:39.722734 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:39.724353 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:39.724358 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:39.726623 (+ 2265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:39.726644 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:39.727608 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:39.727617 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:39.727984 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:39.728795 (+ 811us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:57:39.728801 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:39.728866 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:57:39.728980 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:39.729384 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:39.729397 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:39.729409 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 14:57:39.729616 (+ 207us) client_negotiation.cc:240] Negotiation successful
0928 14:57:39.730087 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":267,"threads_started":1}
I20260928 14:57:39.733570 9181 heartbeater.cc:347] Connected to a master server at 127.31.40.62:40541
I20260928 14:57:39.733970 9181 heartbeater.cc:464] Registering TS with master...
I20260928 14:57:39.735193 9181 heartbeater.cc:511] Master 127.31.40.62:40541 requested a full tablet report, sending...
I20260928 14:57:39.737037 8863 ts_manager.cc:194] Registered new tserver with Master: 711207c18ccb424bb223ad958e45c938 (127.31.40.2:39583)
I20260928 14:57:39.738308 8863 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.40.2:53441
W20260928 14:57:40.072369 9184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:40.072646 9184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:40.072731 9184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:40.079820 9184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:40.080020 9184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:40.080140 9184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.3
I20260928 14:57:40.089366 9184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:33891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.40.3
--webserver_port=0
--tserver_master_addrs=127.31.40.62:40541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.3
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:57:40.090600 9184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:40.092394 9184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:40.106079 9189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:40.106127 9192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:40.106179 9190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:40.109750 9184 server_base.cc:1034] running on GCE node
I20260928 14:57:40.110399 9184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:40.111526 9184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:40.112726 9184 hybrid_clock.cc:648] HybridClock initialized: now 1790607460112637 us; error 83 us; skew 500 ppm
I20260928 14:57:40.113071 9184 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 14:57:40.115394 9184 webserver.cc:460] Webserver started at http://127.31.40.3:46205/ using document root <none> and password file <none>
I20260928 14:57:40.116261 9184 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:40.116351 9184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:40.116734 9184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:40.119692 9184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-2/data/instance:
uuid: "24e767aa18594bb986193b42fef19996"
format_stamp: "Formatted at 2026-09-28 14:57:40 on dist-test-slave-j968"
server_key: "6ffe8ae589b3cb5df60321a8b80a83b9"
server_key_iv: "82a033e9a3b11fd17ad8cdb4b78870e4"
server_key_version: "encryptionkey@0"
I20260928 14:57:40.120668 9184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-2/wal/instance:
uuid: "24e767aa18594bb986193b42fef19996"
format_stamp: "Formatted at 2026-09-28 14:57:40 on dist-test-slave-j968"
server_key: "6ffe8ae589b3cb5df60321a8b80a83b9"
server_key_iv: "82a033e9a3b11fd17ad8cdb4b78870e4"
server_key_version: "encryptionkey@0"
I20260928 14:57:40.126689 9184 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 14:57:40.131125 9198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:40.133431 9184 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 14:57:40.133675 9184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-2/wal
uuid: "24e767aa18594bb986193b42fef19996"
format_stamp: "Formatted at 2026-09-28 14:57:40 on dist-test-slave-j968"
server_key: "6ffe8ae589b3cb5df60321a8b80a83b9"
server_key_iv: "82a033e9a3b11fd17ad8cdb4b78870e4"
server_key_version: "encryptionkey@0"
I20260928 14:57:40.134133 9184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:40.148881 9184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:40.149755 9184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:40.150049 9184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:40.151103 9184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:57:40.152750 9184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:57:40.152872 9184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:40.153016 9184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:57:40.153107 9184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:40.188294 9184 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.3:43313
I20260928 14:57:40.188359 9310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.3:43313 every 8 connection(s)
I20260928 14:57:40.189707 9184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
I20260928 14:57:40.194043 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 9184
I20260928 14:57:40.194211 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-2/wal/instance
I20260928 14:57:40.194718 31904 external_mini_cluster.cc:1428] Setting key 45d4a0cfa399e177dc290b829220a993
I20260928 14:57:40.203512 9052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:40.192525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:40629 (local address 127.31.40.62:40541)
0928 14:57:40.192716 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:40.192720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:40.194153 (+ 1433us) server_negotiation.cc:407] Connection header received
0928 14:57:40.196462 (+ 2309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:40.196467 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:40.196559 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:40.196665 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:40.198046 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:40.198778 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:40.200543 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:40.201033 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:40.201179 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:40.202637 (+ 1458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:40.202641 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:40.202644 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:57:40.202662 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:40.202693 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:57:40.202705 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:40.202811 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:40.203053 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 14:57:40.203154 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 14:57:40.204623 9313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:40.193060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40541 (local address 127.31.40.3:40629)
0928 14:57:40.193981 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:40.194028 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:40.196169 (+ 2141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:40.196906 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:40.196914 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:40.197045 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:40.197885 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:40.197904 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:40.198947 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:40.198951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:40.200392 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:40.200404 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:40.201274 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:40.201281 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:40.201518 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:40.202285 (+ 767us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:57:40.202287 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:40.202333 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:57:40.202436 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:40.202802 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:40.202812 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:40.202821 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 14:57:40.203053 (+ 232us) client_negotiation.cc:240] Negotiation successful
0928 14:57:40.203478 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":271,"threads_started":1}
I20260928 14:57:40.206112 9311 heartbeater.cc:347] Connected to a master server at 127.31.40.62:40541
I20260928 14:57:40.206431 9311 heartbeater.cc:464] Registering TS with master...
I20260928 14:57:40.207149 9311 heartbeater.cc:511] Master 127.31.40.62:40541 requested a full tablet report, sending...
I20260928 14:57:40.208714 8863 ts_manager.cc:194] Registered new tserver with Master: 24e767aa18594bb986193b42fef19996 (127.31.40.3:43313)
I20260928 14:57:40.209627 8863 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.40.3:40629
I20260928 14:57:40.210512 31904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 14:57:40.216991 9052 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:40.215169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51086 (local address 127.31.40.62:40541)
0928 14:57:40.215369 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:40.215377 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:40.215922 (+ 545us) server_negotiation.cc:407] Connection header received
0928 14:57:40.216107 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:40.216110 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:40.216187 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:40.216282 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:40.216674 (+ 392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51086: BlockingRecv error: recv got EOF from 127.0.0.1:51086 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 14:57:40.217796 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 8923
I20260928 14:57:40.226053 9320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:40.218742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60616 (local address 127.31.40.1:40359)
0928 14:57:40.219326 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:40.219333 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:40.219367 (+ 34us) server_negotiation.cc:407] Connection header received
0928 14:57:40.219426 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:40.219432 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:40.219650 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:40.219780 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:40.220215 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:40.221195 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:40.222555 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:40.223197 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:40.223312 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:40.224621 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:40.224629 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:40.224647 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:57:40.225264 (+ 617us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:40.225313 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:57:40.225333 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:40.225457 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:40.225620 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 14:57:40.225734 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":335,"threads_started":1}
I20260928 14:57:40.227406 9045 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:57:40.309105 9050 heartbeater.cc:503] Master 127.31.40.62:40541 was elected leader, sending a full tablet report...
I20260928 14:57:40.310565 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 8923
W20260928 14:57:40.327992 8850 connection.cc:535] server connection from 127.31.40.1:56079 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 14:57:40.328168 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 9054
I20260928 14:57:40.336633 9322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:40.329160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41732 (local address 127.31.40.2:39583)
0928 14:57:40.329664 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:40.329671 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:40.329699 (+ 28us) server_negotiation.cc:407] Connection header received
0928 14:57:40.329769 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:40.329775 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:40.330060 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:40.330200 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:40.330660 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:40.331714 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:40.333156 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:40.333759 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:40.333875 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:40.334942 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:40.334951 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:40.334966 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:57:40.335645 (+ 679us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:40.335699 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:57:40.335718 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:40.335809 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:40.336068 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 14:57:40.336246 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":243,"threads_started":1}
I20260928 14:57:40.338095 9176 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:57:40.421460 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 9054
I20260928 14:57:40.438933 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 9184
I20260928 14:57:40.448057 9324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:40.440182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38330 (local address 127.31.40.3:43313)
0928 14:57:40.440983 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:40.440990 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:40.441024 (+ 34us) server_negotiation.cc:407] Connection header received
0928 14:57:40.441086 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:40.441093 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:40.441404 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:40.441554 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:40.442079 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:40.443285 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:40.444820 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:40.445411 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:40.445512 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:40.446574 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:40.446582 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:40.446596 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:57:40.447202 (+ 606us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:40.447247 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:57:40.447266 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:40.447366 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:40.447502 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 14:57:40.447627 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":442,"threads_started":1}
I20260928 14:57:40.449318 9306 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:57:40.534214 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 9184
I20260928 14:57:40.551216 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 8832
I20260928 14:57:40.552662 8893 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:57:40.656961 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 8832
2026-09-28T14:57:40Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2533 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T14:57:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:57:40Z Disabled control of system clock
I20260928 14:57:40.732254 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:35251
--webserver_interface=127.31.40.62
--webserver_port=0
--builtin_ntp_servers=127.31.40.20:38249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.40.62:35251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 14:57:41.076014 9330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:41.076357 9330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:41.076447 9330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:41.084096 9330 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:57:41.084223 9330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:41.084261 9330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:41.084316 9330 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:57:41.084367 9330 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:57:41.094036 9330 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:38249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.40.62:35251
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:35251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:57:41.095371 9330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:41.097353 9330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:41.109843 9336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:41.109869 9335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:41.109897 9338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:41.110818 9330 server_base.cc:1034] running on GCE node
I20260928 14:57:41.111708 9330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:41.113389 9330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:41.114646 9330 hybrid_clock.cc:648] HybridClock initialized: now 1790607461114597 us; error 44 us; skew 500 ppm
I20260928 14:57:41.115077 9330 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 14:57:41.117708 9330 webserver.cc:460] Webserver started at http://127.31.40.62:41535/ using document root <none> and password file <none>
I20260928 14:57:41.118489 9330 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:41.118631 9330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:41.118978 9330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:41.122329 9330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/master-0/data/instance:
uuid: "ce969af2e73f4472a51a6702794bfdaf"
format_stamp: "Formatted at 2026-09-28 14:57:41 on dist-test-slave-j968"
server_key: "835db16dd4b2b9c010790ef376551a7c"
server_key_iv: "8d65ac290897f2243cd9a3cfba05555c"
server_key_version: "encryptionkey@0"
I20260928 14:57:41.123274 9330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/master-0/wal/instance:
uuid: "ce969af2e73f4472a51a6702794bfdaf"
format_stamp: "Formatted at 2026-09-28 14:57:41 on dist-test-slave-j968"
server_key: "835db16dd4b2b9c010790ef376551a7c"
server_key_iv: "8d65ac290897f2243cd9a3cfba05555c"
server_key_version: "encryptionkey@0"
I20260928 14:57:41.129484 9330 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20260928 14:57:41.133591 9344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:41.135329 9330 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
I20260928 14:57:41.135581 9330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/master-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/master-0/wal
uuid: "ce969af2e73f4472a51a6702794bfdaf"
format_stamp: "Formatted at 2026-09-28 14:57:41 on dist-test-slave-j968"
server_key: "835db16dd4b2b9c010790ef376551a7c"
server_key_iv: "8d65ac290897f2243cd9a3cfba05555c"
server_key_version: "encryptionkey@0"
I20260928 14:57:41.135931 9330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:41.142566 9330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:41.143509 9330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:41.143843 9330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:41.163256 9330 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.62:35251
I20260928 14:57:41.163290 9395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.62:35251 every 8 connection(s)
I20260928 14:57:41.164825 9330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
I20260928 14:57:41.167603 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 9330
I20260928 14:57:41.167831 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/master-0/wal/instance
I20260928 14:57:41.168660 31904 external_mini_cluster.cc:1428] Setting key a9779b47fe9893ea3a5324d95c7f3056
I20260928 14:57:41.169804 9396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:41.178411 9399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:41.170867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42508 (local address 127.31.40.62:35251)
0928 14:57:41.172045 (+ 1178us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:41.172059 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:41.172104 (+ 45us) server_negotiation.cc:407] Connection header received
0928 14:57:41.174452 (+ 2348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:41.174478 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:41.174992 (+ 514us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:41.175364 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:41.176664 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:41.176671 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:41.176686 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:57:41.176701 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:41.176766 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:57:41.176781 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:41.176976 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:41.177143 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 14:57:41.177487 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":416,"threads_started":1}
I20260928 14:57:41.184124 9396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce969af2e73f4472a51a6702794bfdaf: Bootstrap starting.
I20260928 14:57:41.187518 9396 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce969af2e73f4472a51a6702794bfdaf: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:41.188695 9396 log.cc:824] T 00000000000000000000000000000000 P ce969af2e73f4472a51a6702794bfdaf: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:41.191798 9396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce969af2e73f4472a51a6702794bfdaf: No bootstrap required, opened a new log
I20260928 14:57:41.199285 9396 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce969af2e73f4472a51a6702794bfdaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce969af2e73f4472a51a6702794bfdaf" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 35251 } }
I20260928 14:57:41.199730 9396 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce969af2e73f4472a51a6702794bfdaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:41.199831 9396 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce969af2e73f4472a51a6702794bfdaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce969af2e73f4472a51a6702794bfdaf, State: Initialized, Role: FOLLOWER
I20260928 14:57:41.200534 9396 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce969af2e73f4472a51a6702794bfdaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce969af2e73f4472a51a6702794bfdaf" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 35251 } }
I20260928 14:57:41.200796 9396 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce969af2e73f4472a51a6702794bfdaf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:57:41.200937 9396 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce969af2e73f4472a51a6702794bfdaf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:57:41.201133 9396 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce969af2e73f4472a51a6702794bfdaf [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:41.203697 9396 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce969af2e73f4472a51a6702794bfdaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce969af2e73f4472a51a6702794bfdaf" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 35251 } }
I20260928 14:57:41.204273 9396 leader_election.cc:302] T 00000000000000000000000000000000 P ce969af2e73f4472a51a6702794bfdaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce969af2e73f4472a51a6702794bfdaf; no voters:
I20260928 14:57:41.204988 9396 leader_election.cc:288] T 00000000000000000000000000000000 P ce969af2e73f4472a51a6702794bfdaf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:57:41.205122 9401 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce969af2e73f4472a51a6702794bfdaf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:57:41.205940 9401 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce969af2e73f4472a51a6702794bfdaf [term 1 LEADER]: Becoming Leader. State: Replica: ce969af2e73f4472a51a6702794bfdaf, State: Running, Role: LEADER
I20260928 14:57:41.206581 9401 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce969af2e73f4472a51a6702794bfdaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce969af2e73f4472a51a6702794bfdaf" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 35251 } }
I20260928 14:57:41.207238 9396 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce969af2e73f4472a51a6702794bfdaf [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:57:41.209596 9402 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce969af2e73f4472a51a6702794bfdaf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce969af2e73f4472a51a6702794bfdaf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce969af2e73f4472a51a6702794bfdaf" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 35251 } } }
I20260928 14:57:41.210069 9402 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce969af2e73f4472a51a6702794bfdaf [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:41.211552 9403 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce969af2e73f4472a51a6702794bfdaf [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce969af2e73f4472a51a6702794bfdaf. Latest consensus state: current_term: 1 leader_uuid: "ce969af2e73f4472a51a6702794bfdaf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce969af2e73f4472a51a6702794bfdaf" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 35251 } } }
I20260928 14:57:41.212003 9403 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce969af2e73f4472a51a6702794bfdaf [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:41.213248 9410 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:57:41.218743 9410 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:57:41.229004 9410 catalog_manager.cc:1347] Generated new cluster ID: bc43c5f13255462d9d1fd5bccb53080b
I20260928 14:57:41.229168 9410 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:57:41.246140 9410 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:57:41.246578 9410 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:57:41.256811 9410 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce969af2e73f4472a51a6702794bfdaf: Generated new TSK 0
I20260928 14:57:41.257692 9410 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:57:41.272815 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.1:0
--local_ip_for_outbound_sockets=127.31.40.1
--webserver_interface=127.31.40.1
--webserver_port=0
--tserver_master_addrs=127.31.40.62:35251
--builtin_ntp_servers=127.31.40.20:38249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 14:57:41.627512 9420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:41.627969 9420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:41.628158 9420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:41.635535 9420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:41.635659 9420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:41.635851 9420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.1
I20260928 14:57:41.645962 9420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:38249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.40.1
--webserver_port=0
--tserver_master_addrs=127.31.40.62:35251
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.1
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:57:41.647480 9420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:41.649874 9420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:41.664394 9425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:41.664354 9428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:41.664317 9426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:41.665190 9420 server_base.cc:1034] running on GCE node
I20260928 14:57:41.665807 9420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:41.666854 9420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:41.668129 9420 hybrid_clock.cc:648] HybridClock initialized: now 1790607461668032 us; error 66 us; skew 500 ppm
I20260928 14:57:41.668596 9420 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 14:57:41.671137 9420 webserver.cc:460] Webserver started at http://127.31.40.1:46217/ using document root <none> and password file <none>
I20260928 14:57:41.672024 9420 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:41.672173 9420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:41.672585 9420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:41.677958 9420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-0/data/instance:
uuid: "90692412d14f42a7bc1eec9200bb200e"
format_stamp: "Formatted at 2026-09-28 14:57:41 on dist-test-slave-j968"
server_key: "368bd8f4b19b71085347fb132c966e83"
server_key_iv: "524019ef380111ad5773760159c820f9"
server_key_version: "encryptionkey@0"
I20260928 14:57:41.678941 9420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-0/wal/instance:
uuid: "90692412d14f42a7bc1eec9200bb200e"
format_stamp: "Formatted at 2026-09-28 14:57:41 on dist-test-slave-j968"
server_key: "368bd8f4b19b71085347fb132c966e83"
server_key_iv: "524019ef380111ad5773760159c820f9"
server_key_version: "encryptionkey@0"
I20260928 14:57:41.684808 9420 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 14:57:41.688624 9434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:41.690301 9420 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:57:41.690536 9420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-0/wal
uuid: "90692412d14f42a7bc1eec9200bb200e"
format_stamp: "Formatted at 2026-09-28 14:57:41 on dist-test-slave-j968"
server_key: "368bd8f4b19b71085347fb132c966e83"
server_key_iv: "524019ef380111ad5773760159c820f9"
server_key_version: "encryptionkey@0"
I20260928 14:57:41.690819 9420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:41.697286 9420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:41.698275 9420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:41.698624 9420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:41.699744 9420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:57:41.701725 9420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:57:41.701884 9420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:41.702034 9420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:57:41.702140 9420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:41.737879 9420 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.1:38895
I20260928 14:57:41.737941 9546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.1:38895 every 8 connection(s)
I20260928 14:57:41.739428 9420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
I20260928 14:57:41.740378 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 9420
I20260928 14:57:41.740646 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-0/wal/instance
I20260928 14:57:41.741966 31904 external_mini_cluster.cc:1428] Setting key 1ca1f2de9bb15b22796dd13906bc44a9
I20260928 14:57:41.746160 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.2:0
--local_ip_for_outbound_sockets=127.31.40.2
--webserver_interface=127.31.40.2
--webserver_port=0
--tserver_master_addrs=127.31.40.62:35251
--builtin_ntp_servers=127.31.40.20:38249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 14:57:41.752700 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:41.744660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:49335 (local address 127.31.40.62:35251)
0928 14:57:41.745144 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:41.745150 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:41.746880 (+ 1730us) server_negotiation.cc:407] Connection header received
0928 14:57:41.749890 (+ 3010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:41.749896 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:41.749960 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:41.750070 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:41.751851 (+ 1781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:41.751856 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:41.751859 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:57:41.751902 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:41.752000 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:57:41.752015 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:41.752142 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:41.752375 (+ 233us) server_negotiation.cc:299] Negotiation successful
0928 14:57:41.752457 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":237,"threads_started":1}
I20260928 14:57:41.754158 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:41.745452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:35251 (local address 127.31.40.1:49335)
0928 14:57:41.746689 (+ 1237us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:41.746742 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:41.749461 (+ 2719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:41.750406 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:41.750421 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:41.750587 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:41.751403 (+ 816us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:57:41.751409 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:41.751468 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:57:41.751608 (+ 140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:41.752131 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:41.752145 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:41.752157 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 14:57:41.752463 (+ 306us) client_negotiation.cc:240] Negotiation successful
0928 14:57:41.752787 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":327,"threads_started":1}
I20260928 14:57:41.756317 9547 heartbeater.cc:347] Connected to a master server at 127.31.40.62:35251
I20260928 14:57:41.756670 9547 heartbeater.cc:464] Registering TS with master...
I20260928 14:57:41.757339 9547 heartbeater.cc:511] Master 127.31.40.62:35251 requested a full tablet report, sending...
I20260928 14:57:41.759672 9361 ts_manager.cc:194] Registered new tserver with Master: 90692412d14f42a7bc1eec9200bb200e (127.31.40.1:38895)
W20260928 14:57:42.094792 9551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:42.095129 9551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:42.095288 9551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:42.102684 9551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:42.102847 9551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:42.103050 9551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.2
I20260928 14:57:42.112763 9551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:38249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.40.2
--webserver_port=0
--tserver_master_addrs=127.31.40.62:35251
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.2
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:57:42.114199 9551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:42.116079 9551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:42.130154 9556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:42.130209 9557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:42.130218 9559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:42.131217 9551 server_base.cc:1034] running on GCE node
I20260928 14:57:42.132174 9551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:42.133415 9551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:42.134635 9551 hybrid_clock.cc:648] HybridClock initialized: now 1790607462134574 us; error 51 us; skew 500 ppm
I20260928 14:57:42.135025 9551 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 14:57:42.143483 9551 webserver.cc:460] Webserver started at http://127.31.40.2:36985/ using document root <none> and password file <none>
I20260928 14:57:42.144500 9551 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:42.144645 9551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:42.145079 9551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:42.148461 9551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-1/data/instance:
uuid: "bef7325b3b974527ad14c7b9572f6ae4"
format_stamp: "Formatted at 2026-09-28 14:57:42 on dist-test-slave-j968"
server_key: "5ec08214d81b0c51a115a22b9fb0db52"
server_key_iv: "6b13b01233a59727401fe0f3fac53ba5"
server_key_version: "encryptionkey@0"
I20260928 14:57:42.149299 9551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-1/wal/instance:
uuid: "bef7325b3b974527ad14c7b9572f6ae4"
format_stamp: "Formatted at 2026-09-28 14:57:42 on dist-test-slave-j968"
server_key: "5ec08214d81b0c51a115a22b9fb0db52"
server_key_iv: "6b13b01233a59727401fe0f3fac53ba5"
server_key_version: "encryptionkey@0"
I20260928 14:57:42.154656 9551 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 14:57:42.158320 9565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:42.160045 9551 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 14:57:42.160231 9551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-1/wal
uuid: "bef7325b3b974527ad14c7b9572f6ae4"
format_stamp: "Formatted at 2026-09-28 14:57:42 on dist-test-slave-j968"
server_key: "5ec08214d81b0c51a115a22b9fb0db52"
server_key_iv: "6b13b01233a59727401fe0f3fac53ba5"
server_key_version: "encryptionkey@0"
I20260928 14:57:42.160475 9551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:42.166662 9551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:42.167506 9551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:42.167784 9551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:42.168826 9551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:57:42.170280 9551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:57:42.170404 9551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:42.170543 9551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:57:42.170631 9551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:42.205228 9551 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.2:40175
I20260928 14:57:42.205279 9677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.2:40175 every 8 connection(s)
I20260928 14:57:42.206641 9551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
I20260928 14:57:42.214618 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 9551
I20260928 14:57:42.214838 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-1/wal/instance
I20260928 14:57:42.215369 31904 external_mini_cluster.cc:1428] Setting key 74eaa83ef231267b8b3f8801b59af178
I20260928 14:57:42.215453 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:42.209507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:56017 (local address 127.31.40.62:35251)
0928 14:57:42.209685 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:42.209690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:42.211315 (+ 1625us) server_negotiation.cc:407] Connection header received
0928 14:57:42.213103 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:42.213107 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:42.213167 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:42.213266 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:42.214601 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:42.214608 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:42.214613 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:57:42.214634 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:42.214674 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:57:42.214692 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:42.214840 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:42.215097 (+ 257us) server_negotiation.cc:299] Negotiation successful
0928 14:57:42.215171 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 14:57:42.216727 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:42.210116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:35251 (local address 127.31.40.2:56017)
0928 14:57:42.211107 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:42.211150 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:42.212815 (+ 1665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:42.213476 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:42.213484 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:42.213607 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:42.214263 (+ 656us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:57:42.214266 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:42.214304 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:57:42.214397 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:42.214860 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:42.214868 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:42.214874 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 14:57:42.215181 (+ 307us) client_negotiation.cc:240] Negotiation successful
0928 14:57:42.215460 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":253,"threads_started":1}
I20260928 14:57:42.218232 9678 heartbeater.cc:347] Connected to a master server at 127.31.40.62:35251
I20260928 14:57:42.218609 9678 heartbeater.cc:464] Registering TS with master...
I20260928 14:57:42.219079 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.3:0
--local_ip_for_outbound_sockets=127.31.40.3
--webserver_interface=127.31.40.3
--webserver_port=0
--tserver_master_addrs=127.31.40.62:35251
--builtin_ntp_servers=127.31.40.20:38249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 14:57:42.219378 9678 heartbeater.cc:511] Master 127.31.40.62:35251 requested a full tablet report, sending...
I20260928 14:57:42.221035 9361 ts_manager.cc:194] Registered new tserver with Master: bef7325b3b974527ad14c7b9572f6ae4 (127.31.40.2:40175)
W20260928 14:57:42.563374 9681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:42.563778 9681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:42.564013 9681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:42.572647 9681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:42.572762 9681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:42.572893 9681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.3
I20260928 14:57:42.583384 9681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:38249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.40.3
--webserver_port=0
--tserver_master_addrs=127.31.40.62:35251
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.3
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:57:42.584717 9681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:42.586505 9681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:42.600400 9687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:42.600428 9686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:42.600543 9689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:42.601521 9681 server_base.cc:1034] running on GCE node
I20260928 14:57:42.602295 9681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:42.603367 9681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:42.604636 9681 hybrid_clock.cc:648] HybridClock initialized: now 1790607462604562 us; error 68 us; skew 500 ppm
I20260928 14:57:42.605094 9681 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 14:57:42.607676 9681 webserver.cc:460] Webserver started at http://127.31.40.3:35375/ using document root <none> and password file <none>
I20260928 14:57:42.608537 9681 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:42.608639 9681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:42.608997 9681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:42.612226 9681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-2/data/instance:
uuid: "2fe6d00daf2e40db8dfe31c08392fd8b"
format_stamp: "Formatted at 2026-09-28 14:57:42 on dist-test-slave-j968"
server_key: "4bdc0aa15d8f9c48e22ca39de7885362"
server_key_iv: "3ef91aaa6a585772df1f21f875072ac9"
server_key_version: "encryptionkey@0"
I20260928 14:57:42.613088 9681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-2/wal/instance:
uuid: "2fe6d00daf2e40db8dfe31c08392fd8b"
format_stamp: "Formatted at 2026-09-28 14:57:42 on dist-test-slave-j968"
server_key: "4bdc0aa15d8f9c48e22ca39de7885362"
server_key_iv: "3ef91aaa6a585772df1f21f875072ac9"
server_key_version: "encryptionkey@0"
I20260928 14:57:42.618589 9681 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 14:57:42.622615 9695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:42.624517 9681 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 14:57:42.624730 9681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-2/wal
uuid: "2fe6d00daf2e40db8dfe31c08392fd8b"
format_stamp: "Formatted at 2026-09-28 14:57:42 on dist-test-slave-j968"
server_key: "4bdc0aa15d8f9c48e22ca39de7885362"
server_key_iv: "3ef91aaa6a585772df1f21f875072ac9"
server_key_version: "encryptionkey@0"
I20260928 14:57:42.625077 9681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:42.631677 9681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:42.632758 9681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:42.633062 9681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:42.634119 9681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:57:42.635741 9681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:57:42.635975 9681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:42.636129 9681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:57:42.636238 9681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:42.672430 9681 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.3:41367
I20260928 14:57:42.672485 9807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.3:41367 every 8 connection(s)
I20260928 14:57:42.673906 9681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
I20260928 14:57:42.682590 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:42.676657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:44351 (local address 127.31.40.62:35251)
0928 14:57:42.676829 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:42.676833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:42.678315 (+ 1482us) server_negotiation.cc:407] Connection header received
0928 14:57:42.680214 (+ 1899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:42.680219 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:42.680278 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:42.680370 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:42.681748 (+ 1378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:42.681753 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:42.681756 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:57:42.681775 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:42.681904 (+ 129us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:57:42.681919 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:42.682031 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:42.682273 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 14:57:42.682342 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 14:57:42.683480 9810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:42.677159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:35251 (local address 127.31.40.3:44351)
0928 14:57:42.678132 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:42.678176 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:42.679716 (+ 1540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:42.680617 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:42.680626 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:42.680726 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:42.681419 (+ 693us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:57:42.681421 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:42.681460 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:57:42.681550 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:42.682048 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:42.682056 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:42.682061 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:57:42.682279 (+ 218us) client_negotiation.cc:240] Negotiation successful
0928 14:57:42.682489 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":291,"threads_started":1}
I20260928 14:57:42.684125 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 9681
I20260928 14:57:42.684397 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607365362981-31904-0/minicluster-data/ts-2/wal/instance
I20260928 14:57:42.684995 31904 external_mini_cluster.cc:1428] Setting key 61f6208b77a5b662c80689b7cda27948
I20260928 14:57:42.685181 9808 heartbeater.cc:347] Connected to a master server at 127.31.40.62:35251
I20260928 14:57:42.685508 9808 heartbeater.cc:464] Registering TS with master...
I20260928 14:57:42.686211 9808 heartbeater.cc:511] Master 127.31.40.62:35251 requested a full tablet report, sending...
I20260928 14:57:42.687624 9360 ts_manager.cc:194] Registered new tserver with Master: 2fe6d00daf2e40db8dfe31c08392fd8b (127.31.40.3:41367)
I20260928 14:57:42.700851 31904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 14:57:42.707832 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:42.706152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42524 (local address 127.31.40.62:35251)
0928 14:57:42.706334 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:42.706339 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:42.706830 (+ 491us) server_negotiation.cc:407] Connection header received
0928 14:57:42.706989 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:42.706993 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:42.707080 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:42.707180 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:42.707583 (+ 403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42524: BlockingRecv error: recv got EOF from 127.0.0.1:42524 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 14:57:42.708200 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 9420
I20260928 14:57:42.711985 9817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:42.709244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40858 (local address 127.31.40.1:38895)
0928 14:57:42.709763 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:42.709770 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:42.709797 (+ 27us) server_negotiation.cc:407] Connection header received
0928 14:57:42.709856 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:42.709862 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:42.710145 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:42.710312 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:42.711257 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:42.711264 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:42.711278 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:57:42.711320 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:42.711382 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:57:42.711398 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:42.711510 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:42.711586 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 14:57:42.711669 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":250,"threads_started":1}
I20260928 14:57:42.713323 9542 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:57:42.796705 9547 heartbeater.cc:503] Master 127.31.40.62:35251 was elected leader, sending a full tablet report...
I20260928 14:57:42.798626 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 9420
I20260928 14:57:42.813783 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 9551
I20260928 14:57:42.817937 9819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:42.814899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44186 (local address 127.31.40.2:40175)
0928 14:57:42.815510 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:42.815518 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:42.815548 (+ 30us) server_negotiation.cc:407] Connection header received
0928 14:57:42.815613 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:42.815620 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:42.815913 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:42.816157 (+ 244us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:42.817040 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:42.817046 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:42.817059 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:57:42.817077 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:42.817119 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:57:42.817133 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:42.817249 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:42.817488 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 14:57:42.817635 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":301,"threads_started":1}
I20260928 14:57:42.819273 9673 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:57:42.902094 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 9551
I20260928 14:57:42.921610 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 9681
I20260928 14:57:42.925918 9821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:42.922768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44986 (local address 127.31.40.3:41367)
0928 14:57:42.923384 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:42.923391 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:42.923424 (+ 33us) server_negotiation.cc:407] Connection header received
0928 14:57:42.923486 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:42.923495 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:42.923742 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:42.923937 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:42.924903 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:42.924909 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:42.924922 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:57:42.924934 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:42.924973 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:57:42.924991 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:42.925196 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:42.925467 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 14:57:42.925564 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":305,"threads_started":1}
I20260928 14:57:42.927299 9803 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:57:43.011195 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 9681
I20260928 14:57:43.027622 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 9330
I20260928 14:57:43.028903 9391 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:57:43.132489 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 9330
2026-09-28T14:57:43Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2475 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:57:43 dist-test-slave-j968 krb5kdc[9826](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:57:43 dist-test-slave-j968 krb5kdc[9826](info): set up 2 sockets
Sep 28 14:57:43 dist-test-slave-j968 krb5kdc[9826](info): commencing operation
krb5kdc: starting...
W20260928 14:57:44.249874 31904 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Sep 28 14:57:44 dist-test-slave-j968 krb5kdc[9826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:57:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:57:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab.
Entry for principal kudu/127.31.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab.
I20260928 14:57:44.500648 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:39003
--webserver_interface=127.31.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:43935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.40.62:39003
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:57:44.859663 9839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:44.860059 9839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:44.860220 9839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:44.867583 9839 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:57:44.867692 9839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:44.867784 9839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:44.867905 9839 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:57:44.867985 9839 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:57:44.878170 9839 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:43935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.40.62:39003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:39003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:57:44.879624 9839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:44.881009 9839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:44.893241 9844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:44.893283 9845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:44.893445 9847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:44.894567 9839 server_base.cc:1034] running on GCE node
I20260928 14:57:44.895331 9839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:44.896849 9839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:44.898067 9839 hybrid_clock.cc:648] HybridClock initialized: now 1790607464898024 us; error 35 us; skew 500 ppm
I20260928 14:57:44.898388 9839 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:57:44 dist-test-slave-j968 krb5kdc[9826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607464, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:44.901678 9839 init.cc:377] Logged in from keytab as kudu/127.31.40.62@KRBTEST.COM (short username kudu)
I20260928 14:57:44.903272 9839 webserver.cc:460] Webserver started at http://127.31.40.62:39549/ using document root <none> and password file <none>
I20260928 14:57:44.904132 9839 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:44.904244 9839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:44.904510 9839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:44.907537 9839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/master-0/data/instance:
uuid: "43fb19d28476466282dd426bb69dc6b1"
format_stamp: "Formatted at 2026-09-28 14:57:44 on dist-test-slave-j968"
server_key: "1c3960ddcbd2d5eeb6e8ff661b1cf6ba"
server_key_iv: "9b9ff63e95f5b89f17f44a5c1fd6f31a"
server_key_version: "encryptionkey@0"
I20260928 14:57:44.908334 9839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/master-0/wal/instance:
uuid: "43fb19d28476466282dd426bb69dc6b1"
format_stamp: "Formatted at 2026-09-28 14:57:44 on dist-test-slave-j968"
server_key: "1c3960ddcbd2d5eeb6e8ff661b1cf6ba"
server_key_iv: "9b9ff63e95f5b89f17f44a5c1fd6f31a"
server_key_version: "encryptionkey@0"
I20260928 14:57:44.913790 9839 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 14:57:44.917371 9854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:44.918778 9839 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 14:57:44.918928 9839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/master-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/master-0/wal
uuid: "43fb19d28476466282dd426bb69dc6b1"
format_stamp: "Formatted at 2026-09-28 14:57:44 on dist-test-slave-j968"
server_key: "1c3960ddcbd2d5eeb6e8ff661b1cf6ba"
server_key_iv: "9b9ff63e95f5b89f17f44a5c1fd6f31a"
server_key_version: "encryptionkey@0"
I20260928 14:57:44.919124 9839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:44.943400 9839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:44.947659 9839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:44.947934 9839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:44.967110 9839 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.62:39003
I20260928 14:57:44.967133 9905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.62:39003 every 8 connection(s)
I20260928 14:57:44.968587 9839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
I20260928 14:57:44.972774 9906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:44.975189 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 9839
I20260928 14:57:44.975426 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/master-0/wal/instance
I20260928 14:57:44.976263 31904 external_mini_cluster.cc:1428] Setting key 36134af7e1f8ffc49cc2d54c3136dc90
I20260928 14:57:44.986207 9906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43fb19d28476466282dd426bb69dc6b1: Bootstrap starting.
I20260928 14:57:44.990197 9906 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43fb19d28476466282dd426bb69dc6b1: Neither blocks nor log segments found. Creating new log.
Sep 28 14:57:44 dist-test-slave-j968 krb5kdc[9826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:57:44.991403 9906 log.cc:824] T 00000000000000000000000000000000 P 43fb19d28476466282dd426bb69dc6b1: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:44.994776 9906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43fb19d28476466282dd426bb69dc6b1: No bootstrap required, opened a new log
I20260928 14:57:45.000679 9909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:44.978475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47540 (local address 127.31.40.62:39003)
0928 14:57:44.979423 (+ 948us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:44.979435 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:44.979477 (+ 42us) server_negotiation.cc:407] Connection header received
0928 14:57:44.981225 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:44.981247 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:44.981644 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:44.981986 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:44.983824 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:44.985206 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:44.986960 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:44.987650 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:44.987799 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:44.992534 (+ 4735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:44.992549 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:44.992570 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:44.992623 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:44.996062 (+ 3439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:44.996878 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:44.996888 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:44.996895 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:44.997111 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:44.997540 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:44.997547 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:44.997549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:44.998178 (+ 629us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:44.998425 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:44.998897 (+ 472us) server_negotiation.cc:299] Negotiation successful
0928 14:57:44.999381 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":706,"thread_start_us":458,"threads_started":1}
I20260928 14:57:45.004549 9906 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43fb19d28476466282dd426bb69dc6b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43fb19d28476466282dd426bb69dc6b1" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 39003 } }
I20260928 14:57:45.004951 9906 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43fb19d28476466282dd426bb69dc6b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:45.005141 9906 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43fb19d28476466282dd426bb69dc6b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43fb19d28476466282dd426bb69dc6b1, State: Initialized, Role: FOLLOWER
I20260928 14:57:45.005925 9906 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43fb19d28476466282dd426bb69dc6b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43fb19d28476466282dd426bb69dc6b1" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 39003 } }
I20260928 14:57:45.006209 9906 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43fb19d28476466282dd426bb69dc6b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:57:45.006315 9906 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43fb19d28476466282dd426bb69dc6b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:57:45.006443 9906 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43fb19d28476466282dd426bb69dc6b1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:45.008766 9906 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43fb19d28476466282dd426bb69dc6b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43fb19d28476466282dd426bb69dc6b1" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 39003 } }
I20260928 14:57:45.009348 9906 leader_election.cc:302] T 00000000000000000000000000000000 P 43fb19d28476466282dd426bb69dc6b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43fb19d28476466282dd426bb69dc6b1; no voters:
I20260928 14:57:45.010080 9906 leader_election.cc:288] T 00000000000000000000000000000000 P 43fb19d28476466282dd426bb69dc6b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:57:45.010176 9911 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43fb19d28476466282dd426bb69dc6b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:57:45.010927 9911 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43fb19d28476466282dd426bb69dc6b1 [term 1 LEADER]: Becoming Leader. State: Replica: 43fb19d28476466282dd426bb69dc6b1, State: Running, Role: LEADER
I20260928 14:57:45.011591 9911 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43fb19d28476466282dd426bb69dc6b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43fb19d28476466282dd426bb69dc6b1" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 39003 } }
I20260928 14:57:45.012290 9906 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43fb19d28476466282dd426bb69dc6b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:57:45.014514 9912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43fb19d28476466282dd426bb69dc6b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "43fb19d28476466282dd426bb69dc6b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43fb19d28476466282dd426bb69dc6b1" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 39003 } } }
I20260928 14:57:45.014881 9912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43fb19d28476466282dd426bb69dc6b1 [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:45.016556 9918 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:57:45.016177 9913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43fb19d28476466282dd426bb69dc6b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43fb19d28476466282dd426bb69dc6b1. Latest consensus state: current_term: 1 leader_uuid: "43fb19d28476466282dd426bb69dc6b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43fb19d28476466282dd426bb69dc6b1" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 39003 } } }
I20260928 14:57:45.017220 9913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43fb19d28476466282dd426bb69dc6b1 [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:45.022149 9918 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:57:45.032249 9918 catalog_manager.cc:1347] Generated new cluster ID: 319601dc03b14790808f5dfe029f6140
I20260928 14:57:45.032397 9918 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:57:45.050531 9918 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:57:45.052646 9918 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:57:45.060930 9918 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 43fb19d28476466282dd426bb69dc6b1: Generated new TSK 0
I20260928 14:57:45.061862 9918 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.40.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab.
Entry for principal kudu/127.31.40.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab.
I20260928 14:57:45.136401 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.1:0
--local_ip_for_outbound_sockets=127.31.40.1
--webserver_interface=127.31.40.1
--webserver_port=0
--tserver_master_addrs=127.31.40.62:39003
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab
--principal=kudu/127.31.40.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:43935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:57:45.493216 9932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:45.493491 9932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:45.493585 9932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:45.501060 9932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:45.501174 9932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:45.501358 9932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.1
I20260928 14:57:45.511505 9932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:43935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab
--principal=kudu/127.31.40.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.40.1
--webserver_port=0
--tserver_master_addrs=127.31.40.62:39003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.1
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:57:45.512933 9932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:45.514205 9932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:45.528301 9937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:45.528219 9940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:45.528260 9938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:45.529356 9932 server_base.cc:1034] running on GCE node
I20260928 14:57:45.530072 9932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:45.531106 9932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:45.532342 9932 hybrid_clock.cc:648] HybridClock initialized: now 1790607465532256 us; error 68 us; skew 500 ppm
I20260928 14:57:45.532742 9932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:57:45 dist-test-slave-j968 krb5kdc[9826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607465, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:45.536094 9932 init.cc:377] Logged in from keytab as kudu/127.31.40.1@KRBTEST.COM (short username kudu)
I20260928 14:57:45.537678 9932 webserver.cc:460] Webserver started at http://127.31.40.1:41599/ using document root <none> and password file <none>
I20260928 14:57:45.538420 9932 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:45.538566 9932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:45.538883 9932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:45.542001 9932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/ts-0/data/instance:
uuid: "ea97d1977c7f4f03aaf3748a258a6a88"
format_stamp: "Formatted at 2026-09-28 14:57:45 on dist-test-slave-j968"
server_key: "ed1734a8f339d00d753d529b5fb54937"
server_key_iv: "5f99fc11dfe326158acb7b7b01c4502d"
server_key_version: "encryptionkey@0"
I20260928 14:57:45.542829 9932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/ts-0/wal/instance:
uuid: "ea97d1977c7f4f03aaf3748a258a6a88"
format_stamp: "Formatted at 2026-09-28 14:57:45 on dist-test-slave-j968"
server_key: "ed1734a8f339d00d753d529b5fb54937"
server_key_iv: "5f99fc11dfe326158acb7b7b01c4502d"
server_key_version: "encryptionkey@0"
I20260928 14:57:45.548518 9932 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 14:57:45.552445 9947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:45.553952 9932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 14:57:45.554169 9932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/ts-0/wal
uuid: "ea97d1977c7f4f03aaf3748a258a6a88"
format_stamp: "Formatted at 2026-09-28 14:57:45 on dist-test-slave-j968"
server_key: "ed1734a8f339d00d753d529b5fb54937"
server_key_iv: "5f99fc11dfe326158acb7b7b01c4502d"
server_key_version: "encryptionkey@0"
I20260928 14:57:45.554356 9932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:45.570057 9932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:45.575112 9932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:45.575410 9932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:45.576562 9932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:57:45.578013 9932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:57:45.578094 9932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:45.578194 9932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:57:45.578248 9932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:45.614288 9932 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.1:36611
I20260928 14:57:45.614327 10059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.1:36611 every 8 connection(s)
I20260928 14:57:45.615731 9932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
I20260928 14:57:45.625784 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 9932
I20260928 14:57:45.626017 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/ts-0/wal/instance
I20260928 14:57:45.626822 31904 external_mini_cluster.cc:1428] Setting key c73d1e82d913fa275f1778b1759f631d
Sep 28 14:57:45 dist-test-slave-j968 krb5kdc[9826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607465, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.1@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:57:45.638401 10062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:45.618544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:35549 (local address 127.31.40.62:39003)
0928 14:57:45.619056 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:45.619062 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:45.620177 (+ 1115us) server_negotiation.cc:407] Connection header received
0928 14:57:45.621968 (+ 1791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:45.621973 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:45.622041 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:45.622140 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:45.623969 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:45.624772 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:45.627168 (+ 2396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:45.627761 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:45.628000 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:45.632451 (+ 4451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:45.632456 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:45.632459 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:45.632521 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:45.635187 (+ 2666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:45.635991 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:45.635999 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:45.636004 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:45.636137 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:45.636595 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:45.636600 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:45.636604 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:45.636883 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:45.637050 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:45.637792 (+ 742us) server_negotiation.cc:299] Negotiation successful
0928 14:57:45.638031 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":284,"threads_started":1}
I20260928 14:57:45.639475 10063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:45.619204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:39003 (local address 127.31.40.1:35549)
0928 14:57:45.620019 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:45.620066 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:45.621687 (+ 1621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:45.622401 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:45.622410 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:45.622990 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:45.623709 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:45.623730 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:45.624985 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:45.624988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:45.626989 (+ 2001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:45.627016 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:45.628168 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:45.628174 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:45.628413 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:45.629255 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:45.629288 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:45.632200 (+ 2912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:45.635359 (+ 3159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:45.635369 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:45.635392 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:45.635834 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:45.636259 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:45.636264 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:45.636268 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:45.636472 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:45.637028 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:45.637037 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:45.637407 (+ 370us) client_negotiation.cc:769] Sending connection context
0928 14:57:45.637684 (+ 277us) client_negotiation.cc:240] Negotiation successful
0928 14:57:45.638285 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":316,"threads_started":1}
I20260928 14:57:45.641264 10060 heartbeater.cc:347] Connected to a master server at 127.31.40.62:39003
I20260928 14:57:45.641649 10060 heartbeater.cc:464] Registering TS with master...
I20260928 14:57:45.642699 10060 heartbeater.cc:511] Master 127.31.40.62:39003 requested a full tablet report, sending...
I20260928 14:57:45.645449 9871 ts_manager.cc:194] Registered new tserver with Master: ea97d1977c7f4f03aaf3748a258a6a88 (127.31.40.1:36611)
I20260928 14:57:45.647961 9871 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.1@KRBTEST.COM'} at 127.31.40.1:35549
WARNING: no policy specified for kudu/127.31.40.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab.
Entry for principal kudu/127.31.40.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab.
I20260928 14:57:45.686982 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.2:0
--local_ip_for_outbound_sockets=127.31.40.2
--webserver_interface=127.31.40.2
--webserver_port=0
--tserver_master_addrs=127.31.40.62:39003
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab
--principal=kudu/127.31.40.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:43935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:57:46.036912 10066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:46.037227 10066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:46.037326 10066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:46.044911 10066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:46.045054 10066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:46.045173 10066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.2
I20260928 14:57:46.056131 10066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:43935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab
--principal=kudu/127.31.40.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.40.2
--webserver_port=0
--tserver_master_addrs=127.31.40.62:39003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.2
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:57:46.057629 10066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:46.059123 10066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:46.073449 10071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:46.073484 10072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:46.073520 10074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:46.074558 10066 server_base.cc:1034] running on GCE node
I20260928 14:57:46.075330 10066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:46.076709 10066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:46.078015 10066 hybrid_clock.cc:648] HybridClock initialized: now 1790607466077952 us; error 55 us; skew 500 ppm
I20260928 14:57:46.078393 10066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:57:46 dist-test-slave-j968 krb5kdc[9826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607466, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:46.082120 10066 init.cc:377] Logged in from keytab as kudu/127.31.40.2@KRBTEST.COM (short username kudu)
I20260928 14:57:46.083762 10066 webserver.cc:460] Webserver started at http://127.31.40.2:41731/ using document root <none> and password file <none>
I20260928 14:57:46.084995 10066 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:46.085180 10066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:46.085568 10066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:46.088967 10066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/ts-1/data/instance:
uuid: "aaa586fafb524ec3a2e685fbb648a119"
format_stamp: "Formatted at 2026-09-28 14:57:46 on dist-test-slave-j968"
server_key: "138bb122879fc407de58a55dbe11ffb8"
server_key_iv: "7e560961a5cea1b357e11512293a1fce"
server_key_version: "encryptionkey@0"
I20260928 14:57:46.089815 10066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/ts-1/wal/instance:
uuid: "aaa586fafb524ec3a2e685fbb648a119"
format_stamp: "Formatted at 2026-09-28 14:57:46 on dist-test-slave-j968"
server_key: "138bb122879fc407de58a55dbe11ffb8"
server_key_iv: "7e560961a5cea1b357e11512293a1fce"
server_key_version: "encryptionkey@0"
I20260928 14:57:46.095669 10066 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 14:57:46.099697 10081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:46.101300 10066 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:57:46.101508 10066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/ts-1/wal
uuid: "aaa586fafb524ec3a2e685fbb648a119"
format_stamp: "Formatted at 2026-09-28 14:57:46 on dist-test-slave-j968"
server_key: "138bb122879fc407de58a55dbe11ffb8"
server_key_iv: "7e560961a5cea1b357e11512293a1fce"
server_key_version: "encryptionkey@0"
I20260928 14:57:46.101737 10066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:46.128428 10066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:46.135177 10066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:46.135614 10066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:46.137220 10066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:57:46.139644 10066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:57:46.139763 10066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:46.139933 10066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:57:46.140023 10066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260928 14:57:46.175747 10066 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.2:39175
I20260928 14:57:46.175846 10193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.2:39175 every 8 connection(s)
I20260928 14:57:46.177088 10066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
I20260928 14:57:46.182626 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 10066
I20260928 14:57:46.182826 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/ts-1/wal/instance
I20260928 14:57:46.183295 31904 external_mini_cluster.cc:1428] Setting key 39a19b08adb5ee2df4728f77943bd592
Sep 28 14:57:46 dist-test-slave-j968 krb5kdc[9826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607466, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.2@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:57:46.200044 10196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:46.179803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:38873 (local address 127.31.40.62:39003)
0928 14:57:46.180351 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:46.180358 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:46.181477 (+ 1119us) server_negotiation.cc:407] Connection header received
0928 14:57:46.183615 (+ 2138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:46.183621 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:46.183725 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:46.183853 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:46.185854 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:46.186669 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:46.188593 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:46.189083 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:46.189232 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:46.193414 (+ 4182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:46.193420 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:46.193424 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:46.193464 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:46.196549 (+ 3085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:46.197433 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:46.197439 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:46.197441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:46.197585 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:46.198064 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:46.198074 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:46.198076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:46.198365 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:46.198532 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:46.199319 (+ 787us) server_negotiation.cc:299] Negotiation successful
0928 14:57:46.199574 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":309,"threads_started":1}
I20260928 14:57:46.201192 10197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:46.180488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:39003 (local address 127.31.40.2:38873)
0928 14:57:46.181308 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:46.181355 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:46.183261 (+ 1906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:46.184172 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:46.184181 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:46.184973 (+ 792us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:46.185676 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:46.185696 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:46.186860 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:46.186864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:46.188424 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:46.188438 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:46.189340 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:46.189349 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:46.189619 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:46.190387 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:46.190413 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:46.193083 (+ 2670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:46.196732 (+ 3649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:46.196743 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:46.196766 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:46.197280 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:46.197710 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:46.197715 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:46.197719 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:46.197936 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:46.198520 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:46.198529 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:46.198917 (+ 388us) client_negotiation.cc:769] Sending connection context
0928 14:57:46.199209 (+ 292us) client_negotiation.cc:240] Negotiation successful
0928 14:57:46.199913 (+ 704us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":360,"threads_started":1}
I20260928 14:57:46.203127 10194 heartbeater.cc:347] Connected to a master server at 127.31.40.62:39003
I20260928 14:57:46.203490 10194 heartbeater.cc:464] Registering TS with master...
I20260928 14:57:46.204341 10194 heartbeater.cc:511] Master 127.31.40.62:39003 requested a full tablet report, sending...
I20260928 14:57:46.206278 9871 ts_manager.cc:194] Registered new tserver with Master: aaa586fafb524ec3a2e685fbb648a119 (127.31.40.2:39175)
WARNING: no policy specified for kudu/127.31.40.3@KRBTEST.COM; defaulting to no policy
I20260928 14:57:46.207794 9871 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.2@KRBTEST.COM'} at 127.31.40.2:38873
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab.
Entry for principal kudu/127.31.40.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab.
I20260928 14:57:46.240144 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.3:0
--local_ip_for_outbound_sockets=127.31.40.3
--webserver_interface=127.31.40.3
--webserver_port=0
--tserver_master_addrs=127.31.40.62:39003
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab
--principal=kudu/127.31.40.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:43935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:57:46.587248 10200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:46.587598 10200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:46.587769 10200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:46.595791 10200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:46.595984 10200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:46.596163 10200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.3
I20260928 14:57:46.606966 10200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:43935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab
--principal=kudu/127.31.40.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.40.3
--webserver_port=0
--tserver_master_addrs=127.31.40.62:39003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.3
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:57:46.608438 10200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:46.609817 10200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:46.624366 10205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:46.624315 10206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:46.624787 10208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:46.625223 10200 server_base.cc:1034] running on GCE node
I20260928 14:57:46.625828 10200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:46.626823 10200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:46.628106 10200 hybrid_clock.cc:648] HybridClock initialized: now 1790607466628035 us; error 54 us; skew 500 ppm
I20260928 14:57:46.628432 10200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:57:46 dist-test-slave-j968 krb5kdc[9826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607466, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:46.631821 10200 init.cc:377] Logged in from keytab as kudu/127.31.40.3@KRBTEST.COM (short username kudu)
I20260928 14:57:46.633486 10200 webserver.cc:460] Webserver started at http://127.31.40.3:34701/ using document root <none> and password file <none>
I20260928 14:57:46.634234 10200 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:46.634378 10200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:46.634685 10200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:46.637740 10200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/ts-2/data/instance:
uuid: "2b029142fcca4355b8eefb6f60917952"
format_stamp: "Formatted at 2026-09-28 14:57:46 on dist-test-slave-j968"
server_key: "a2efcb1e824da752c4b72a024bc3c4b0"
server_key_iv: "a4d66303234db63c372dac48f7806724"
server_key_version: "encryptionkey@0"
I20260928 14:57:46.638541 10200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/ts-2/wal/instance:
uuid: "2b029142fcca4355b8eefb6f60917952"
format_stamp: "Formatted at 2026-09-28 14:57:46 on dist-test-slave-j968"
server_key: "a2efcb1e824da752c4b72a024bc3c4b0"
server_key_iv: "a4d66303234db63c372dac48f7806724"
server_key_version: "encryptionkey@0"
I20260928 14:57:46.644299 10200 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 14:57:46.648098 10215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:46.649564 10200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:57:46.649757 10200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/ts-2/wal
uuid: "2b029142fcca4355b8eefb6f60917952"
format_stamp: "Formatted at 2026-09-28 14:57:46 on dist-test-slave-j968"
server_key: "a2efcb1e824da752c4b72a024bc3c4b0"
server_key_iv: "a4d66303234db63c372dac48f7806724"
server_key_version: "encryptionkey@0"
I20260928 14:57:46.649964 10200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:46.651193 10060 heartbeater.cc:503] Master 127.31.40.62:39003 was elected leader, sending a full tablet report...
I20260928 14:57:46.664436 10200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:46.668902 10200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:46.669214 10200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:46.670178 10200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:57:46.671537 10200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:57:46.671636 10200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:46.671785 10200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:57:46.671859 10200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:46.706017 10200 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.3:42149
I20260928 14:57:46.706046 10327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.3:42149 every 8 connection(s)
I20260928 14:57:46.707345 10200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
I20260928 14:57:46.716089 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 10200
I20260928 14:57:46.716287 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/ts-2/wal/instance
I20260928 14:57:46.716804 31904 external_mini_cluster.cc:1428] Setting key 88c5e134a8678d78ee9d002861e9ee9a
Sep 28 14:57:46 dist-test-slave-j968 krb5kdc[9826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607466, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.3@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:57:46.728588 10330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:46.710085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:50653 (local address 127.31.40.62:39003)
0928 14:57:46.710583 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:46.710588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:46.711636 (+ 1048us) server_negotiation.cc:407] Connection header received
0928 14:57:46.713598 (+ 1962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:46.713603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:46.713725 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:46.713839 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:46.715368 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:46.716576 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:46.718683 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:46.719243 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:46.719338 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:46.723141 (+ 3803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:46.723146 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:46.723149 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:46.723181 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:46.725850 (+ 2669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:46.726532 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:46.726536 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:46.726538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:46.726650 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:46.727041 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:46.727044 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:46.727046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:46.727316 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:46.727429 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:46.728162 (+ 733us) server_negotiation.cc:299] Negotiation successful
0928 14:57:46.728322 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":224,"threads_started":1}
I20260928 14:57:46.729609 10331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:46.710745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:39003 (local address 127.31.40.3:50653)
0928 14:57:46.711467 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:46.711518 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:46.713332 (+ 1814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:46.714088 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:46.714096 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:46.714611 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:46.715200 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:46.715217 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:46.716731 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:46.716736 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:46.718521 (+ 1785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:46.718536 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:46.719497 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:46.719506 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:46.719825 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:46.720538 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:46.720562 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:46.722920 (+ 2358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:46.725990 (+ 3070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:46.726000 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:46.726013 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:46.726432 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:46.726736 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:46.726739 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:46.726742 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:46.726917 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:46.727420 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:46.727427 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:46.727773 (+ 346us) client_negotiation.cc:769] Sending connection context
0928 14:57:46.728066 (+ 293us) client_negotiation.cc:240] Negotiation successful
0928 14:57:46.728461 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":281,"threads_started":1}
I20260928 14:57:46.731340 10328 heartbeater.cc:347] Connected to a master server at 127.31.40.62:39003
I20260928 14:57:46.731690 10328 heartbeater.cc:464] Registering TS with master...
I20260928 14:57:46.732607 10328 heartbeater.cc:511] Master 127.31.40.62:39003 requested a full tablet report, sending...
I20260928 14:57:46.734043 9871 ts_manager.cc:194] Registered new tserver with Master: 2b029142fcca4355b8eefb6f60917952 (127.31.40.3:42149)
I20260928 14:57:46.734861 9871 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.3@KRBTEST.COM'} at 127.31.40.3:50653
I20260928 14:57:46.745282 31904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:57:46.763938 10330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:46.749859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47552 (local address 127.31.40.62:39003)
0928 14:57:46.750087 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:46.750092 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:46.750534 (+ 442us) server_negotiation.cc:407] Connection header received
0928 14:57:46.750684 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:46.750687 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:46.750752 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:46.750835 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:46.752416 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:46.753155 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:46.754468 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:46.755116 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:46.755224 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:46.757496 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:46.757501 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:46.757505 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:46.757539 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:46.760795 (+ 3256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:46.761745 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:46.761751 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:46.761755 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:46.761905 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:46.762285 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:46.762291 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:46.762294 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:46.762723 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:46.762970 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:46.763326 (+ 356us) server_negotiation.cc:299] Negotiation successful
0928 14:57:46.763563 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 14:57:46.768147 9871 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47552:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:57:46.775111 9871 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:57:46.812012 10341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:46.789907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.2:39175 (local address 127.0.0.1:37984)
0928 14:57:46.801129 (+ 11222us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:46.801151 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:46.801283 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:46.802818 (+ 1535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:46.802830 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:46.802875 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:46.803449 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:46.803473 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:46.807126 (+ 3653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:46.807132 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:46.808983 (+ 1851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:46.808996 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:46.810851 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:46.810857 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:46.811070 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:46.811137 (+ 67us) client_negotiation.cc:769] Sending connection context
0928 14:57:46.811301 (+ 164us) client_negotiation.cc:240] Negotiation successful
0928 14:57:46.811531 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11065,"mutex_wait_us":5,"thread_start_us":10051,"threads_started":1}
I20260928 14:57:46.813027 10344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:46.790230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37984 (local address 127.31.40.2:39175)
0928 14:57:46.801960 (+ 11730us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:46.801974 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:46.802013 (+ 39us) server_negotiation.cc:407] Connection header received
0928 14:57:46.802141 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:46.802153 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:46.802509 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:46.803740 (+ 1231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:46.804085 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:46.806983 (+ 2898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:46.809139 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:46.810603 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:46.810744 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:46.810948 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:46.811347 (+ 399us) server_negotiation.cc:299] Negotiation successful
0928 14:57:46.811533 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11471,"thread_start_us":11702,"threads_started":1}
I20260928 14:57:46.813912 10346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:46.802368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:36611 (local address 127.0.0.1:35800)
0928 14:57:46.805895 (+ 3527us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:46.805915 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:46.806043 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:46.806900 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:46.806905 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:46.807023 (+ 118us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:46.807610 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:46.807623 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:46.809516 (+ 1893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:46.809520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:46.811220 (+ 1700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:46.811235 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:46.813124 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:46.813130 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:46.813302 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:46.813346 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 14:57:46.813450 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 14:57:46.813593 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3410,"thread_start_us":3463,"threads_started":1}
I20260928 14:57:46.814548 10345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:46.801530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35800 (local address 127.31.40.1:36611)
0928 14:57:46.805784 (+ 4254us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:46.805790 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:46.806062 (+ 272us) server_negotiation.cc:407] Connection header received
0928 14:57:46.806172 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:46.806179 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:46.806436 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:46.806631 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:46.807822 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:46.809349 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:46.811390 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:46.812904 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:46.813066 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:46.813304 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:46.813875 (+ 571us) server_negotiation.cc:299] Negotiation successful
0928 14:57:46.814189 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4092,"thread_start_us":3933,"threads_started":1}
I20260928 14:57:46.814675 10342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:46.789504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44084 (local address 127.31.40.3:42149)
0928 14:57:46.805910 (+ 16406us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:46.805920 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:46.805971 (+ 51us) server_negotiation.cc:407] Connection header received
0928 14:57:46.806072 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:46.806083 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:46.806347 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:46.806624 (+ 277us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:46.807604 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:46.809325 (+ 1721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:46.811718 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:46.813340 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:46.813482 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:46.813765 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:46.814154 (+ 389us) server_negotiation.cc:299] Negotiation successful
0928 14:57:46.814313 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16220,"thread_start_us":16594,"threads_started":1}
I20260928 14:57:46.814582 10343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:46.789225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.3:42149 (local address 127.0.0.1:44084)
0928 14:57:46.800410 (+ 11185us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:46.800460 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:46.800641 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:46.806858 (+ 6217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:46.806864 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:46.806889 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:46.807420 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:46.807430 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:46.809463 (+ 2033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:46.809468 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:46.811526 (+ 2058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:46.811540 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:46.813573 (+ 2033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:46.813580 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:46.813774 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:46.813817 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 14:57:46.813906 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 14:57:46.814218 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10917,"spinlock_wait_cycles":120448,"thread_start_us":11575,"threads_started":1}
I20260928 14:57:46.817377 10129 tablet_service.cc:1467] Processing CreateTablet for tablet a04ca4c403c844629293aecd91a1a57e (DEFAULT_TABLE table=test-table [id=dbde0b22de5b40938a011889214f9cb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:57:46.818125 9995 tablet_service.cc:1467] Processing CreateTablet for tablet a04ca4c403c844629293aecd91a1a57e (DEFAULT_TABLE table=test-table [id=dbde0b22de5b40938a011889214f9cb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:57:46.818953 10129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a04ca4c403c844629293aecd91a1a57e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:46.818738 10263 tablet_service.cc:1467] Processing CreateTablet for tablet a04ca4c403c844629293aecd91a1a57e (DEFAULT_TABLE table=test-table [id=dbde0b22de5b40938a011889214f9cb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:57:46.820127 10263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a04ca4c403c844629293aecd91a1a57e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:46.820370 9995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a04ca4c403c844629293aecd91a1a57e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:46.831533 10347 tablet_bootstrap.cc:492] T a04ca4c403c844629293aecd91a1a57e P aaa586fafb524ec3a2e685fbb648a119: Bootstrap starting.
I20260928 14:57:46.835129 10347 tablet_bootstrap.cc:654] T a04ca4c403c844629293aecd91a1a57e P aaa586fafb524ec3a2e685fbb648a119: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:46.836215 10349 tablet_bootstrap.cc:492] T a04ca4c403c844629293aecd91a1a57e P 2b029142fcca4355b8eefb6f60917952: Bootstrap starting.
I20260928 14:57:46.836656 10347 log.cc:824] T a04ca4c403c844629293aecd91a1a57e P aaa586fafb524ec3a2e685fbb648a119: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:46.837195 10348 tablet_bootstrap.cc:492] T a04ca4c403c844629293aecd91a1a57e P ea97d1977c7f4f03aaf3748a258a6a88: Bootstrap starting.
I20260928 14:57:46.841194 10347 tablet_bootstrap.cc:492] T a04ca4c403c844629293aecd91a1a57e P aaa586fafb524ec3a2e685fbb648a119: No bootstrap required, opened a new log
I20260928 14:57:46.841228 10349 tablet_bootstrap.cc:654] T a04ca4c403c844629293aecd91a1a57e P 2b029142fcca4355b8eefb6f60917952: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:46.841235 10348 tablet_bootstrap.cc:654] T a04ca4c403c844629293aecd91a1a57e P ea97d1977c7f4f03aaf3748a258a6a88: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:46.841504 10347 ts_tablet_manager.cc:1397] T a04ca4c403c844629293aecd91a1a57e P aaa586fafb524ec3a2e685fbb648a119: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.001s
I20260928 14:57:46.842413 10348 log.cc:824] T a04ca4c403c844629293aecd91a1a57e P ea97d1977c7f4f03aaf3748a258a6a88: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:46.842489 10349 log.cc:824] T a04ca4c403c844629293aecd91a1a57e P 2b029142fcca4355b8eefb6f60917952: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:46.845692 10349 tablet_bootstrap.cc:492] T a04ca4c403c844629293aecd91a1a57e P 2b029142fcca4355b8eefb6f60917952: No bootstrap required, opened a new log
I20260928 14:57:46.845824 10348 tablet_bootstrap.cc:492] T a04ca4c403c844629293aecd91a1a57e P ea97d1977c7f4f03aaf3748a258a6a88: No bootstrap required, opened a new log
I20260928 14:57:46.846035 10349 ts_tablet_manager.cc:1397] T a04ca4c403c844629293aecd91a1a57e P 2b029142fcca4355b8eefb6f60917952: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260928 14:57:46.846135 10348 ts_tablet_manager.cc:1397] T a04ca4c403c844629293aecd91a1a57e P ea97d1977c7f4f03aaf3748a258a6a88: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 14:57:46.852722 10347 raft_consensus.cc:348] T a04ca4c403c844629293aecd91a1a57e P aaa586fafb524ec3a2e685fbb648a119 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b029142fcca4355b8eefb6f60917952" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 42149 } } peers { permanent_uuid: "ea97d1977c7f4f03aaf3748a258a6a88" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36611 } } peers { permanent_uuid: "aaa586fafb524ec3a2e685fbb648a119" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39175 } }
I20260928 14:57:46.853258 10347 raft_consensus.cc:374] T a04ca4c403c844629293aecd91a1a57e P aaa586fafb524ec3a2e685fbb648a119 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:46.853358 10347 raft_consensus.cc:729] T a04ca4c403c844629293aecd91a1a57e P aaa586fafb524ec3a2e685fbb648a119 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaa586fafb524ec3a2e685fbb648a119, State: Initialized, Role: FOLLOWER
I20260928 14:57:46.855719 10349 raft_consensus.cc:348] T a04ca4c403c844629293aecd91a1a57e P 2b029142fcca4355b8eefb6f60917952 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b029142fcca4355b8eefb6f60917952" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 42149 } } peers { permanent_uuid: "ea97d1977c7f4f03aaf3748a258a6a88" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36611 } } peers { permanent_uuid: "aaa586fafb524ec3a2e685fbb648a119" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39175 } }
I20260928 14:57:46.856383 10349 raft_consensus.cc:374] T a04ca4c403c844629293aecd91a1a57e P 2b029142fcca4355b8eefb6f60917952 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:46.856510 10349 raft_consensus.cc:729] T a04ca4c403c844629293aecd91a1a57e P 2b029142fcca4355b8eefb6f60917952 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b029142fcca4355b8eefb6f60917952, State: Initialized, Role: FOLLOWER
I20260928 14:57:46.856223 10347 consensus_queue.cc:260] T a04ca4c403c844629293aecd91a1a57e P aaa586fafb524ec3a2e685fbb648a119 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b029142fcca4355b8eefb6f60917952" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 42149 } } peers { permanent_uuid: "ea97d1977c7f4f03aaf3748a258a6a88" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36611 } } peers { permanent_uuid: "aaa586fafb524ec3a2e685fbb648a119" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39175 } }
I20260928 14:57:46.857236 10349 consensus_queue.cc:260] T a04ca4c403c844629293aecd91a1a57e P 2b029142fcca4355b8eefb6f60917952 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b029142fcca4355b8eefb6f60917952" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 42149 } } peers { permanent_uuid: "ea97d1977c7f4f03aaf3748a258a6a88" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36611 } } peers { permanent_uuid: "aaa586fafb524ec3a2e685fbb648a119" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39175 } }
I20260928 14:57:46.857265 10348 raft_consensus.cc:348] T a04ca4c403c844629293aecd91a1a57e P ea97d1977c7f4f03aaf3748a258a6a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b029142fcca4355b8eefb6f60917952" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 42149 } } peers { permanent_uuid: "ea97d1977c7f4f03aaf3748a258a6a88" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36611 } } peers { permanent_uuid: "aaa586fafb524ec3a2e685fbb648a119" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39175 } }
I20260928 14:57:46.857883 10348 raft_consensus.cc:374] T a04ca4c403c844629293aecd91a1a57e P ea97d1977c7f4f03aaf3748a258a6a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:46.858022 10348 raft_consensus.cc:729] T a04ca4c403c844629293aecd91a1a57e P ea97d1977c7f4f03aaf3748a258a6a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea97d1977c7f4f03aaf3748a258a6a88, State: Initialized, Role: FOLLOWER
I20260928 14:57:46.858718 10348 consensus_queue.cc:260] T a04ca4c403c844629293aecd91a1a57e P ea97d1977c7f4f03aaf3748a258a6a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b029142fcca4355b8eefb6f60917952" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 42149 } } peers { permanent_uuid: "ea97d1977c7f4f03aaf3748a258a6a88" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36611 } } peers { permanent_uuid: "aaa586fafb524ec3a2e685fbb648a119" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39175 } }
I20260928 14:57:46.861712 10348 ts_tablet_manager.cc:1428] T a04ca4c403c844629293aecd91a1a57e P ea97d1977c7f4f03aaf3748a258a6a88: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260928 14:57:46.862843 10194 heartbeater.cc:503] Master 127.31.40.62:39003 was elected leader, sending a full tablet report...
I20260928 14:57:46.864070 10347 ts_tablet_manager.cc:1428] T a04ca4c403c844629293aecd91a1a57e P aaa586fafb524ec3a2e685fbb648a119: Time spent starting tablet: real 0.022s user 0.012s sys 0.008s
I20260928 14:57:46.865216 10328 heartbeater.cc:503] Master 127.31.40.62:39003 was elected leader, sending a full tablet report...
I20260928 14:57:46.866837 10349 ts_tablet_manager.cc:1428] T a04ca4c403c844629293aecd91a1a57e P 2b029142fcca4355b8eefb6f60917952: Time spent starting tablet: real 0.021s user 0.012s sys 0.007s
W20260928 14:57:46.868275 10061 tablet.cc:2369] T a04ca4c403c844629293aecd91a1a57e P ea97d1977c7f4f03aaf3748a258a6a88: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:57:46.883592 10354 raft_consensus.cc:482] T a04ca4c403c844629293aecd91a1a57e P 2b029142fcca4355b8eefb6f60917952 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:57:46.884040 10354 raft_consensus.cc:504] T a04ca4c403c844629293aecd91a1a57e P 2b029142fcca4355b8eefb6f60917952 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b029142fcca4355b8eefb6f60917952" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 42149 } } peers { permanent_uuid: "ea97d1977c7f4f03aaf3748a258a6a88" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36611 } } peers { permanent_uuid: "aaa586fafb524ec3a2e685fbb648a119" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39175 } }
I20260928 14:57:46.885860 10354 leader_election.cc:288] T a04ca4c403c844629293aecd91a1a57e P 2b029142fcca4355b8eefb6f60917952 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea97d1977c7f4f03aaf3748a258a6a88 (127.31.40.1:36611), aaa586fafb524ec3a2e685fbb648a119 (127.31.40.2:39175)
I20260928 14:57:46.892740 10345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:46.886316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:44623 (local address 127.31.40.1:36611)
0928 14:57:46.886535 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:46.886541 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:46.886667 (+ 126us) server_negotiation.cc:407] Connection header received
0928 14:57:46.886907 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:46.886915 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:46.887007 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:46.887115 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:46.887550 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:46.888575 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:46.890685 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:46.891911 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:46.892035 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:46.892113 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:46.892459 (+ 346us) server_negotiation.cc:299] Negotiation successful
0928 14:57:46.892556 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 14:57:46.892889 10331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:46.886348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:36611 (local address 127.31.40.3:44623)
0928 14:57:46.886559 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:46.886579 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:46.886733 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:46.887152 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:46.887157 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:46.887189 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:46.887355 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:46.887363 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:46.888699 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:46.888704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:46.890533 (+ 1829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:46.890545 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:46.892073 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:46.892080 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:46.892295 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:46.892331 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 14:57:46.892441 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 14:57:46.892602 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 14:57:46.893791 10344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:46.886316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:40477 (local address 127.31.40.2:39175)
0928 14:57:46.886536 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:46.886542 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:46.887438 (+ 896us) server_negotiation.cc:407] Connection header received
0928 14:57:46.887600 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:46.887604 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:46.887661 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:46.887737 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:46.888399 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:46.889423 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:46.891686 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:46.892930 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:46.893071 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:46.893181 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:46.893510 (+ 329us) server_negotiation.cc:299] Negotiation successful
0928 14:57:46.893615 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 14:57:46.893970 10356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:46.886348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.2:39175 (local address 127.31.40.3:40477)
0928 14:57:46.887340 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:46.887355 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:46.887471 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:46.887778 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:46.887783 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:46.887808 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:46.888205 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:46.888216 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:46.889613 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:46.889618 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:46.891513 (+ 1895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:46.891529 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:46.893183 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:46.893191 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:46.893366 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:46.893389 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 14:57:46.893478 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 14:57:46.893612 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":864,"mutex_wait_us":216,"thread_start_us":392,"threads_started":1}
I20260928 14:57:46.894043 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a04ca4c403c844629293aecd91a1a57e" candidate_uuid: "2b029142fcca4355b8eefb6f60917952" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea97d1977c7f4f03aaf3748a258a6a88" is_pre_election: true
I20260928 14:57:46.894672 10015 raft_consensus.cc:2454] T a04ca4c403c844629293aecd91a1a57e P ea97d1977c7f4f03aaf3748a258a6a88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b029142fcca4355b8eefb6f60917952 in term 0.
I20260928 14:57:46.894953 10149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a04ca4c403c844629293aecd91a1a57e" candidate_uuid: "2b029142fcca4355b8eefb6f60917952" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaa586fafb524ec3a2e685fbb648a119" is_pre_election: true
I20260928 14:57:46.895367 10219 leader_election.cc:302] T a04ca4c403c844629293aecd91a1a57e P 2b029142fcca4355b8eefb6f60917952 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b029142fcca4355b8eefb6f60917952, ea97d1977c7f4f03aaf3748a258a6a88; no voters:
I20260928 14:57:46.895550 10149 raft_consensus.cc:2454] T a04ca4c403c844629293aecd91a1a57e P aaa586fafb524ec3a2e685fbb648a119 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b029142fcca4355b8eefb6f60917952 in term 0.
I20260928 14:57:46.895793 10354 raft_consensus.cc:2781] T a04ca4c403c844629293aecd91a1a57e P 2b029142fcca4355b8eefb6f60917952 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:57:46.895967 10354 raft_consensus.cc:482] T a04ca4c403c844629293aecd91a1a57e P 2b029142fcca4355b8eefb6f60917952 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:57:46.896106 10354 raft_consensus.cc:3037] T a04ca4c403c844629293aecd91a1a57e P 2b029142fcca4355b8eefb6f60917952 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:46.898339 10354 raft_consensus.cc:504] T a04ca4c403c844629293aecd91a1a57e P 2b029142fcca4355b8eefb6f60917952 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b029142fcca4355b8eefb6f60917952" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 42149 } } peers { permanent_uuid: "ea97d1977c7f4f03aaf3748a258a6a88" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36611 } } peers { permanent_uuid: "aaa586fafb524ec3a2e685fbb648a119" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39175 } }
I20260928 14:57:46.898980 10354 leader_election.cc:288] T a04ca4c403c844629293aecd91a1a57e P 2b029142fcca4355b8eefb6f60917952 [CANDIDATE]: Term 1 election: Requested vote from peers ea97d1977c7f4f03aaf3748a258a6a88 (127.31.40.1:36611), aaa586fafb524ec3a2e685fbb648a119 (127.31.40.2:39175)
I20260928 14:57:46.899427 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a04ca4c403c844629293aecd91a1a57e" candidate_uuid: "2b029142fcca4355b8eefb6f60917952" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea97d1977c7f4f03aaf3748a258a6a88"
I20260928 14:57:46.899519 10149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a04ca4c403c844629293aecd91a1a57e" candidate_uuid: "2b029142fcca4355b8eefb6f60917952" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaa586fafb524ec3a2e685fbb648a119"
I20260928 14:57:46.899678 10015 raft_consensus.cc:3037] T a04ca4c403c844629293aecd91a1a57e P ea97d1977c7f4f03aaf3748a258a6a88 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:46.899708 10149 raft_consensus.cc:3037] T a04ca4c403c844629293aecd91a1a57e P aaa586fafb524ec3a2e685fbb648a119 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:46.903105 10015 raft_consensus.cc:2454] T a04ca4c403c844629293aecd91a1a57e P ea97d1977c7f4f03aaf3748a258a6a88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b029142fcca4355b8eefb6f60917952 in term 1.
I20260928 14:57:46.903101 10149 raft_consensus.cc:2454] T a04ca4c403c844629293aecd91a1a57e P aaa586fafb524ec3a2e685fbb648a119 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b029142fcca4355b8eefb6f60917952 in term 1.
I20260928 14:57:46.903534 10219 leader_election.cc:302] T a04ca4c403c844629293aecd91a1a57e P 2b029142fcca4355b8eefb6f60917952 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b029142fcca4355b8eefb6f60917952, ea97d1977c7f4f03aaf3748a258a6a88; no voters:
I20260928 14:57:46.903903 10354 raft_consensus.cc:2781] T a04ca4c403c844629293aecd91a1a57e P 2b029142fcca4355b8eefb6f60917952 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:57:46.904413 10354 raft_consensus.cc:686] T a04ca4c403c844629293aecd91a1a57e P 2b029142fcca4355b8eefb6f60917952 [term 1 LEADER]: Becoming Leader. State: Replica: 2b029142fcca4355b8eefb6f60917952, State: Running, Role: LEADER
I20260928 14:57:46.904983 10354 consensus_queue.cc:237] T a04ca4c403c844629293aecd91a1a57e P 2b029142fcca4355b8eefb6f60917952 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b029142fcca4355b8eefb6f60917952" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 42149 } } peers { permanent_uuid: "ea97d1977c7f4f03aaf3748a258a6a88" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36611 } } peers { permanent_uuid: "aaa586fafb524ec3a2e685fbb648a119" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39175 } }
I20260928 14:57:46.909926 9871 catalog_manager.cc:5495] T a04ca4c403c844629293aecd91a1a57e P 2b029142fcca4355b8eefb6f60917952 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b029142fcca4355b8eefb6f60917952 (127.31.40.3). New cstate: current_term: 1 leader_uuid: "2b029142fcca4355b8eefb6f60917952" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b029142fcca4355b8eefb6f60917952" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 42149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea97d1977c7f4f03aaf3748a258a6a88" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aaa586fafb524ec3a2e685fbb648a119" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39175 } health_report { overall_health: UNKNOWN } } }
I20260928 14:57:46.926745 10342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:46.920957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44096 (local address 127.31.40.3:42149)
0928 14:57:46.921181 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:46.921187 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:46.921258 (+ 71us) server_negotiation.cc:407] Connection header received
0928 14:57:46.921435 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:46.921439 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:46.921513 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:46.921636 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:57:46.921998 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:46.922887 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:46.924416 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:46.924866 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:46.925052 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:46.925565 (+ 513us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:57:46.926168 (+ 603us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:57:46.926294 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:46.926361 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 14:57:46.926437 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260928 14:57:46.929070 10195 tablet.cc:2369] T a04ca4c403c844629293aecd91a1a57e P aaa586fafb524ec3a2e685fbb648a119: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:57:46.935664 10015 raft_consensus.cc:1261] T a04ca4c403c844629293aecd91a1a57e P ea97d1977c7f4f03aaf3748a258a6a88 [term 1 FOLLOWER]: Refusing update from remote peer 2b029142fcca4355b8eefb6f60917952: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:57:46.935664 10149 raft_consensus.cc:1261] T a04ca4c403c844629293aecd91a1a57e P aaa586fafb524ec3a2e685fbb648a119 [term 1 FOLLOWER]: Refusing update from remote peer 2b029142fcca4355b8eefb6f60917952: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:57:46.936892 10357 consensus_queue.cc:1035] T a04ca4c403c844629293aecd91a1a57e P 2b029142fcca4355b8eefb6f60917952 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa586fafb524ec3a2e685fbb648a119" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 39175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:57:46.937287 10354 consensus_queue.cc:1035] T a04ca4c403c844629293aecd91a1a57e P 2b029142fcca4355b8eefb6f60917952 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea97d1977c7f4f03aaf3748a258a6a88" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 36611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:57:46.948117 10360 mvcc.cc:205] Tried to move back new op lower bound from 7334328184556879872 to 7334328184447242240. Current Snapshot: MvccSnapshot[applied={T|T < 7334328184556879872}]
I20260928 14:57:46.949265 10362 mvcc.cc:205] Tried to move back new op lower bound from 7334328184556879872 to 7334328184447242240. Current Snapshot: MvccSnapshot[applied={T|T < 7334328184556879872}]
I20260928 14:57:46.964412 9869 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:57:46.964735 9869 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:57:46.968772 9869 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 43fb19d28476466282dd426bb69dc6b1: Sending DeleteTablet for 3 replicas of tablet a04ca4c403c844629293aecd91a1a57e
I20260928 14:57:46.970172 9995 tablet_service.cc:1514] Processing DeleteTablet for tablet a04ca4c403c844629293aecd91a1a57e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:57:46 UTC) from {username='kudu', principal='kudu/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:35800
I20260928 14:57:46.970184 10129 tablet_service.cc:1514] Processing DeleteTablet for tablet a04ca4c403c844629293aecd91a1a57e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:57:46 UTC) from {username='kudu', principal='kudu/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:37984
I20260928 14:57:46.970193 10263 tablet_service.cc:1514] Processing DeleteTablet for tablet a04ca4c403c844629293aecd91a1a57e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:57:46 UTC) from {username='kudu', principal='kudu/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:44084
I20260928 14:57:46.972371 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 9932
I20260928 14:57:46.990605 10372 tablet_replica.cc:330] T a04ca4c403c844629293aecd91a1a57e P ea97d1977c7f4f03aaf3748a258a6a88: stopping tablet replica
I20260928 14:57:46.991284 10372 raft_consensus.cc:2229] T a04ca4c403c844629293aecd91a1a57e P ea97d1977c7f4f03aaf3748a258a6a88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:57:46.991989 10372 raft_consensus.cc:2258] T a04ca4c403c844629293aecd91a1a57e P ea97d1977c7f4f03aaf3748a258a6a88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:57:46.992908 10371 tablet_replica.cc:330] T a04ca4c403c844629293aecd91a1a57e P aaa586fafb524ec3a2e685fbb648a119: stopping tablet replica
I20260928 14:57:46.993289 10371 raft_consensus.cc:2229] T a04ca4c403c844629293aecd91a1a57e P aaa586fafb524ec3a2e685fbb648a119 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:57:46.993733 10371 raft_consensus.cc:2258] T a04ca4c403c844629293aecd91a1a57e P aaa586fafb524ec3a2e685fbb648a119 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:57:46.998744 10371 ts_tablet_manager.cc:1905] T a04ca4c403c844629293aecd91a1a57e P aaa586fafb524ec3a2e685fbb648a119: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:57:46.999451 10370 tablet_replica.cc:330] T a04ca4c403c844629293aecd91a1a57e P 2b029142fcca4355b8eefb6f60917952: stopping tablet replica
I20260928 14:57:46.999939 10370 raft_consensus.cc:2229] T a04ca4c403c844629293aecd91a1a57e P 2b029142fcca4355b8eefb6f60917952 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:57:47.000682 10370 raft_consensus.cc:2258] T a04ca4c403c844629293aecd91a1a57e P 2b029142fcca4355b8eefb6f60917952 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:57:47.001597 10372 ts_tablet_manager.cc:1905] T a04ca4c403c844629293aecd91a1a57e P ea97d1977c7f4f03aaf3748a258a6a88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:57:47.007421 10371 ts_tablet_manager.cc:1918] T a04ca4c403c844629293aecd91a1a57e P aaa586fafb524ec3a2e685fbb648a119: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:57:47.007623 10371 log.cc:1196] T a04ca4c403c844629293aecd91a1a57e P aaa586fafb524ec3a2e685fbb648a119: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/ts-1/wal/wals/a04ca4c403c844629293aecd91a1a57e
I20260928 14:57:47.008193 10371 ts_tablet_manager.cc:1939] T a04ca4c403c844629293aecd91a1a57e P aaa586fafb524ec3a2e685fbb648a119: Deleting consensus metadata
Sep 28 14:57:47 dist-test-slave-j968 krb5kdc[9826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.1@KRBTEST.COM
I20260928 14:57:47.009613 10372 ts_tablet_manager.cc:1918] T a04ca4c403c844629293aecd91a1a57e P ea97d1977c7f4f03aaf3748a258a6a88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:57:47.009886 10372 log.cc:1196] T a04ca4c403c844629293aecd91a1a57e P ea97d1977c7f4f03aaf3748a258a6a88: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/ts-0/wal/wals/a04ca4c403c844629293aecd91a1a57e
I20260928 14:57:47.010401 9855 catalog_manager.cc:4841] TS aaa586fafb524ec3a2e685fbb648a119 (127.31.40.2:39175): tablet a04ca4c403c844629293aecd91a1a57e (table test-table [id=dbde0b22de5b40938a011889214f9cb7]) successfully deleted
I20260928 14:57:47.010926 10372 ts_tablet_manager.cc:1939] T a04ca4c403c844629293aecd91a1a57e P ea97d1977c7f4f03aaf3748a258a6a88: Deleting consensus metadata
I20260928 14:57:47.012984 10370 ts_tablet_manager.cc:1905] T a04ca4c403c844629293aecd91a1a57e P 2b029142fcca4355b8eefb6f60917952: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:57:47.013789 9858 catalog_manager.cc:4841] TS ea97d1977c7f4f03aaf3748a258a6a88 (127.31.40.1:36611): tablet a04ca4c403c844629293aecd91a1a57e (table test-table [id=dbde0b22de5b40938a011889214f9cb7]) successfully deleted
I20260928 14:57:47.017598 10345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:46.973842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35802 (local address 127.31.40.1:36611)
0928 14:57:46.974024 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:46.974029 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:46.997182 (+ 23153us) server_negotiation.cc:407] Connection header received
0928 14:57:46.997597 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:46.997604 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:46.997698 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:46.997768 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:47.000251 (+ 2483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:47.001355 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:47.003268 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:47.003990 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:47.004110 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:47.009731 (+ 5621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:47.009748 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:47.009766 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:47.009818 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:47.013896 (+ 4078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:47.014915 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:47.014924 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:47.014928 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:47.015106 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:47.015581 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:47.015587 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:47.015591 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:47.016093 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:47.016382 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:47.016730 (+ 348us) server_negotiation.cc:299] Negotiation successful
0928 14:57:47.017002 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 14:57:47.018296 10055 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:57:47.021234 10370 ts_tablet_manager.cc:1918] T a04ca4c403c844629293aecd91a1a57e P 2b029142fcca4355b8eefb6f60917952: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:57:47.021448 10370 log.cc:1196] T a04ca4c403c844629293aecd91a1a57e P 2b029142fcca4355b8eefb6f60917952: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607365362981-31904-0/minicluster-data/ts-2/wal/wals/a04ca4c403c844629293aecd91a1a57e
I20260928 14:57:47.022152 10370 ts_tablet_manager.cc:1939] T a04ca4c403c844629293aecd91a1a57e P 2b029142fcca4355b8eefb6f60917952: Deleting consensus metadata
I20260928 14:57:47.024446 9858 catalog_manager.cc:4841] TS 2b029142fcca4355b8eefb6f60917952 (127.31.40.3:42149): tablet a04ca4c403c844629293aecd91a1a57e (table test-table [id=dbde0b22de5b40938a011889214f9cb7]) successfully deleted
I20260928 14:57:47.152802 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 9932
I20260928 14:57:47.171190 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 10066
Sep 28 14:57:47 dist-test-slave-j968 krb5kdc[9826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.2@KRBTEST.COM
I20260928 14:57:47.190608 10344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:47.172010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37988 (local address 127.31.40.2:39175)
0928 14:57:47.172200 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:47.172206 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:47.172227 (+ 21us) server_negotiation.cc:407] Connection header received
0928 14:57:47.172404 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:47.172411 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:47.172507 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:47.172611 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:47.173837 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:47.174969 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:47.176985 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:47.177696 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:47.177834 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:47.183239 (+ 5405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:47.183252 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:47.183270 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:47.183326 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:47.187234 (+ 3908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:47.188229 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:47.188239 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:47.188244 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:47.188409 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:47.188825 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:47.188830 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:47.188834 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:47.189333 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:47.189624 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:47.190014 (+ 390us) server_negotiation.cc:299] Negotiation successful
0928 14:57:47.190243 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 14:57:47.191303 10189 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:57:47.306782 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 10066
I20260928 14:57:47.325870 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 10200
Sep 28 14:57:47 dist-test-slave-j968 krb5kdc[9826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.3@KRBTEST.COM
I20260928 14:57:47.346107 10342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:47.326710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44108 (local address 127.31.40.3:42149)
0928 14:57:47.326893 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:47.326897 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:47.326953 (+ 56us) server_negotiation.cc:407] Connection header received
0928 14:57:47.327138 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:47.327142 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:47.327205 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:47.327288 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:47.329148 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:47.329964 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:47.331981 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:47.332443 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:47.332577 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:47.338143 (+ 5566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:47.338163 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:47.338186 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:47.338254 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:47.342627 (+ 4373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:47.343499 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:47.343510 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:47.343513 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:47.343771 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:47.344383 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:47.344389 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:47.344393 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:47.344834 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:47.345178 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:47.345455 (+ 277us) server_negotiation.cc:299] Negotiation successful
0928 14:57:47.345679 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 14:57:47.346912 10323 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:57:47.471592 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 10200
I20260928 14:57:47.492820 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 9839
I20260928 14:57:47.494009 9901 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:57:47.604171 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 9839
2026-09-28T14:57:47Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4472 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:57:47 dist-test-slave-j968 krb5kdc[10380](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:57:47 dist-test-slave-j968 krb5kdc[10380](info): set up 2 sockets
Sep 28 14:57:47 dist-test-slave-j968 krb5kdc[10380](info): commencing operation
krb5kdc: starting...
W20260928 14:57:48.785964 31904 mini_kdc.cc:121] Time spent starting KDC: real 1.138s user 0.002s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Sep 28 14:57:48 dist-test-slave-j968 krb5kdc[10380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:57:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:57:48Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.31.40.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.31.40.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.31.40.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/krb5kdc/rangeradmin_127.31.40.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.31.40.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/krb5kdc/rangeradmin_127.31.40.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.31.40.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.31.40.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.31.40.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/krb5kdc/rangerlookup_127.31.40.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.31.40.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/krb5kdc/rangerlookup_127.31.40.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.40.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.31.40.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.40.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/krb5kdc/HTTP_127.31.40.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.40.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/krb5kdc/HTTP_127.31.40.20@KRBTEST.COM.keytab.
I20260928 14:57:49.148432 31904 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskBx3nvY/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/postgres -l logfile start
2026-09-28 14:57:51.899 UTC [10410] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 14:57:51.899 UTC [10410] LOG: listening on IPv4 address "127.31.40.20", port 35673
2026-09-28 14:57:51.903 UTC [10410] LOG: listening on Unix socket "/tmp/.s.PGSQL.35673"
2026-09-28 14:57:51.927 UTC [10413] LOG: database system was shut down at 2026-09-28 14:57:50 UTC
2026-09-28 14:57:51.931 UTC [10410] LOG: database system is ready to accept connections
I20260928 14:57:52.917434 31904 mini_postgres.cc:96] Postgres bound to 35673
2026-09-28 14:57:52.936 UTC [10421] FATAL: database "slave" does not exist
127.31.40.20:35673 - accepting connections
I20260928 14:57:52.937119 31904 mini_ranger.cc:161] Starting Ranger...
I20260928 14:57:52.964929 31904 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 14:57:53.480834 31904 mini_ranger.cc:87] Created ranger Postgres database
I20260928 14:57:53.480996 31904 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ranger-admin
2026-09-28 14:57:53,779 [I] DB FLAVOR :POSTGRES
2026-09-28 14:57:53,780 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 14:57:53,780 [I] Checking connection..
2026-09-28 14:57:53,780 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBx3nvY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBx3nvY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.40.20:35673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 14:57:54,056 [I] Checking connection passed.
2026-09-28 14:57:54,056 [I] --------- Verifying version history table ---------
2026-09-28 14:57:54,057 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBx3nvY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBx3nvY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.40.20:35673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 14:57:54,316 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 14:57:54,317 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBx3nvY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBx3nvY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.40.20:35673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 14:57:54,560 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 14:57:54,560 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 14:57:54,560 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBx3nvY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBx3nvY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.40.20:35673/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBx3nvY/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 14:57:54.829 UTC [10485] WARNING: there is no transaction in progress
2026-09-28 14:57:54,837 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 14:57:54,837 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBx3nvY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBx3nvY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.40.20:35673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 14:57:55,099 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 14:57:55,099 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 14:57:55,100 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBx3nvY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBx3nvY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.40.20:35673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 14:57:55,391 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBx3nvY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBx3nvY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.40.20:35673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 14:57:55,637 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBx3nvY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBx3nvY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.40.20:35673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-j968.c.gcp-upstream.internal','N') ;"
2026-09-28 14:57:55,881 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBx3nvY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBx3nvY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.40.20:35673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 14:57:56,166 [I] Table x_portal_user does not exist in database ranger
2026-09-28 14:57:56,166 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBx3nvY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBx3nvY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.40.20:35673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 14:57:56,439 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 14:57:56,439 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBx3nvY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBx3nvY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.40.20:35673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 14:57:56,683 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBx3nvY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBx3nvY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.40.20:35673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 14:57:56,950 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 14:57:56,950 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBx3nvY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBx3nvY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.40.20:35673/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBx3nvY/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 14:57:58.014 UTC [10594] WARNING: there is no transaction in progress
2026-09-28 14:57:58.033 UTC [10594] WARNING: there is no transaction in progress
2026-09-28 14:57:58.050 UTC [10594] WARNING: there is no transaction in progress
2026-09-28 14:57:58.065 UTC [10594] WARNING: there is no transaction in progress
2026-09-28 14:57:58.083 UTC [10594] WARNING: there is no transaction in progress
2026-09-28 14:57:58.099 UTC [10594] WARNING: there is no transaction in progress
2026-09-28 14:57:58.161 UTC [10594] WARNING: there is no transaction in progress
2026-09-28 14:57:58.168 UTC [10594] WARNING: there is no transaction in progress
2026-09-28 14:57:58.184 UTC [10594] WARNING: there is no transaction in progress
2026-09-28 14:57:58.196 UTC [10594] WARNING: there is no transaction in progress
2026-09-28 14:57:58.207 UTC [10594] WARNING: there is no transaction in progress
2026-09-28 14:57:58.221 UTC [10594] WARNING: there is no transaction in progress
2026-09-28 14:57:58.231 UTC [10594] WARNING: there is no transaction in progress
2026-09-28 14:57:58.239 UTC [10594] WARNING: there is no transaction in progress
2026-09-28 14:57:58.249 UTC [10594] WARNING: there is no transaction in progress
2026-09-28 14:57:58.257 UTC [10594] WARNING: there is no transaction in progress
2026-09-28 14:57:58,998 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 14:57:58,998 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBx3nvY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBx3nvY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.40.20:35673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 14:57:59,250 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 14:57:59,250 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBx3nvY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBx3nvY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.40.20:35673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 14:57:59,520 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 14:57:59,521 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBx3nvY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBx3nvY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.40.20:35673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 14:57:59,798 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 14:57:59,799 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBx3nvY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBx3nvY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.40.20:35673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 14:58:00,079 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBx3nvY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBx3nvY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.40.20:35673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 14:58:00,373 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBx3nvY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBx3nvY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.40.20:35673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-j968.c.gcp-upstream.internal';"
2026-09-28 14:58:00,644 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 14:58:00,644 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBx3nvY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBx3nvY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.40.20:35673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 14:58:00,900 [I] DB_PATCHES have already been applied
I20260928 14:58:00.906490 31904 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ranger-admin:/tmp/dist-test-taskBx3nvY/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskBx3nvY/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskBx3nvY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBx3nvY/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 14:58:00.906579 31904 mini_ranger.cc:193] Using host: 127.31.40.20
I20260928 14:58:00.923051 31904 mini_ranger.cc:227] Ranger admin URL: http://127.31.40.20:53141
Sep 28, 2026 2:58:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:58:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:58:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 2:58:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 2:58:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 2:58:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 2:58:01 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 14:58:02.138401 31904 mini_ranger.cc:160] Time spent starting Ranger: real 9.201s user 0.005s sys 0.039s
Sep 28, 2026 2:58:23 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 2:58:23 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 28, 2026 2:58:23 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 2:58:23 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 2:58:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 2:58:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 2:58:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 2:58:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 2:58:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 2:58:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 2:58:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 2:58:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 2:58:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 2:58:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 2:58:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 2:58:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 2:58:23 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 2:58:24 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 14:58:24.831489 31904 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.31.40.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.31.40.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.31.40.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/krb5kdc/rangerkms_127.31.40.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.31.40.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/krb5kdc/rangerkms_127.31.40.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.40.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.31.40.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.40.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/krb5kdc/HTTP_127.31.40.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.40.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/krb5kdc/HTTP_127.31.40.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 14:58:25 dist-test-slave-j968 krb5kdc[10380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607505, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 14:58:25.020413 31904 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 14:58:25.043360 31904 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 14:58:26.053150 31904 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 14:58:26.053284 31904 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ranger-kms
I20260928 14:58:26.053450 31904 mini_ranger_kms.cc:227] Using postgres at 127.31.40.20:35673
2026-09-28 14:58:26,493 [I] DB FLAVOR :POSTGRES
2026-09-28 14:58:26,493 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 14:58:26,493 [I] Checking connection
2026-09-28 14:58:26,493 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBx3nvY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.40.20:35673/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 14:58:26,729 [I] connection success
2026-09-28 14:58:26,730 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 14:58:26,730 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 14:58:26,730 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBx3nvY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.40.20:35673/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 14:58:26,980 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 14:58:26,980 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 14:58:26,980 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 14:58:26,980 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBx3nvY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.40.20:35673/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 14:58:27,244 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 14:58:27.760771 31904 mini_ranger_kms.cc:310] Created kms service
I20260928 14:58:27.966411 31904 mini_ranger_kms.cc:326] Created kudu user
I20260928 14:58:28.013451 31904 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 14:58:28 dist-test-slave-j968 krb5kdc[10380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607505, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.31.40.20@KRBTEST.COM
I20260928 14:58:28.420111 31904 mini_ranger_kms.cc:384] Added ranger policy
I20260928 14:58:28.420467 31904 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ranger-kms:/tmp/dist-test-taskBx3nvY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBx3nvY/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskBx3nvY/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 14:58:28.420513 31904 mini_ranger_kms.cc:240] Using host: 127.31.40.20
I20260928 14:58:28.443130 31904 mini_ranger_kms.cc:276] Ranger KMS PID: 10775
I20260928 14:58:28.443341 31904 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.31.40.20:35655
Sep 28, 2026 2:58:29 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:58:29 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:58:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 2:58:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 2:58:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 2:58:29 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 28 14:58:30 dist-test-slave-j968 krb5kdc[10380](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790607510, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.40.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:30.800722 31904 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.780s user 0.139s sys 0.321s
I20260928 14:58:30.800998 31904 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 14:58:30.801290 31904 mini_ranger_kms.cc:401] 127.31.40.20:35655/kms/v1/keys
Sep 28 14:58:30 dist-test-slave-j968 krb5kdc[10380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607510, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.40.20@KRBTEST.COM for HTTP/127.31.40.20@KRBTEST.COM
Sep 28 14:58:31 dist-test-slave-j968 krb5kdc[10380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607510, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.40.20@KRBTEST.COM for HTTP/127.31.40.20@KRBTEST.COM
Sep 28 14:58:33 dist-test-slave-j968 krb5kdc[10380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.31.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab.
Entry for principal kudu/127.31.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab.
I20260928 14:58:33.712944 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:32879
--webserver_interface=127.31.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:41653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.40.62:32879
--ranger_config_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.40.20:35655/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:58:34.100677 10825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:34.101039 10825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:34.101191 10825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:34.109225 10825 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:58:34.109392 10825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:34.109506 10825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:34.109602 10825 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:58:34.109710 10825 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:58:34.120623 10825 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:41653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/master-0/wal
--ranger_kms_url=127.31.40.20:35655/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.31.40.62:32879
--ranger_config_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:32879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:58:34.122110 10825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:34.123476 10825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:34.137604 10830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:34.137616 10831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:34.138976 10825 server_base.cc:1034] running on GCE node
W20260928 14:58:34.139124 10833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:34.140270 10825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:34.142413 10825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:34.143708 10825 hybrid_clock.cc:648] HybridClock initialized: now 1790607514143646 us; error 56 us; skew 500 ppm
I20260928 14:58:34.144152 10825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:58:34 dist-test-slave-j968 krb5kdc[10380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607514, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:34.153510 10825 init.cc:377] Logged in from keytab as kudu/127.31.40.62@KRBTEST.COM (short username kudu)
I20260928 14:58:34.155208 10825 webserver.cc:460] Webserver started at http://127.31.40.62:39995/ using document root <none> and password file <none>
I20260928 14:58:34.155987 10825 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:34.156111 10825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:34.156432 10825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 14:58:34 dist-test-slave-j968 krb5kdc[10380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607514, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.62@KRBTEST.COM for HTTP/127.31.40.20@KRBTEST.COM
I20260928 14:58:34.209194 10825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/master-0/data/instance:
uuid: "a3b972e6f1bd40e38fc47fa4ec552e2a"
format_stamp: "Formatted at 2026-09-28 14:58:34 on dist-test-slave-j968"
server_key: "b2e93547946fee794181c53e3e22e439"
server_key_iv: "6a210f6cbd8d9f7489547a98d01d6b5c"
server_key_version: "kuduclusterkey@0"
I20260928 14:58:34.210103 10825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/master-0/wal/instance:
uuid: "a3b972e6f1bd40e38fc47fa4ec552e2a"
format_stamp: "Formatted at 2026-09-28 14:58:34 on dist-test-slave-j968"
server_key: "b2e93547946fee794181c53e3e22e439"
server_key_iv: "6a210f6cbd8d9f7489547a98d01d6b5c"
server_key_version: "kuduclusterkey@0"
I20260928 14:58:34.215968 10825 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260928 14:58:34.238921 10840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:34.240639 10825 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260928 14:58:34.240866 10825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/master-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/master-0/wal
uuid: "a3b972e6f1bd40e38fc47fa4ec552e2a"
format_stamp: "Formatted at 2026-09-28 14:58:34 on dist-test-slave-j968"
server_key: "b2e93547946fee794181c53e3e22e439"
server_key_iv: "6a210f6cbd8d9f7489547a98d01d6b5c"
server_key_version: "kuduclusterkey@0"
I20260928 14:58:34.241101 10825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:34.266258 10825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:34.270519 10825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:34.270795 10825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:34.292017 10825 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.62:32879
I20260928 14:58:34.292161 10891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.62:32879 every 8 connection(s)
I20260928 14:58:34.293675 10825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
I20260928 14:58:34.297580 10892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:34.302346 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 10825
I20260928 14:58:34.302618 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/master-0/wal/instance
I20260928 14:58:34.303350 31904 external_mini_cluster.cc:1428] Setting key 98c31f6dbe45c4536babef141408ce13
I20260928 14:58:34.309821 10892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3b972e6f1bd40e38fc47fa4ec552e2a: Bootstrap starting.
I20260928 14:58:34.313688 10892 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3b972e6f1bd40e38fc47fa4ec552e2a: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:34.314779 10892 log.cc:824] T 00000000000000000000000000000000 P a3b972e6f1bd40e38fc47fa4ec552e2a: Log is configured to *not* fsync() on all Append() calls
I20260928 14:58:34.318064 10892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3b972e6f1bd40e38fc47fa4ec552e2a: No bootstrap required, opened a new log
Sep 28 14:58:34 dist-test-slave-j968 krb5kdc[10380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:58:34.326469 10892 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3b972e6f1bd40e38fc47fa4ec552e2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3b972e6f1bd40e38fc47fa4ec552e2a" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32879 } }
I20260928 14:58:34.326900 10892 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3b972e6f1bd40e38fc47fa4ec552e2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:34.327023 10892 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3b972e6f1bd40e38fc47fa4ec552e2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3b972e6f1bd40e38fc47fa4ec552e2a, State: Initialized, Role: FOLLOWER
I20260928 14:58:34.327728 10892 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3b972e6f1bd40e38fc47fa4ec552e2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3b972e6f1bd40e38fc47fa4ec552e2a" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32879 } }
I20260928 14:58:34.328066 10892 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3b972e6f1bd40e38fc47fa4ec552e2a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:58:34.328220 10892 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3b972e6f1bd40e38fc47fa4ec552e2a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:58:34.328389 10895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:34.305748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42360 (local address 127.31.40.62:32879)
0928 14:58:34.306882 (+ 1134us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:34.306895 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:34.306947 (+ 52us) server_negotiation.cc:407] Connection header received
0928 14:58:34.309230 (+ 2283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:34.309256 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:34.309687 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:34.310177 (+ 490us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:34.311609 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:34.313497 (+ 1888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:34.314961 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:34.316128 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:34.316349 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:34.320618 (+ 4269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:34.320629 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:34.320645 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:34.320687 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:34.323854 (+ 3167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:34.324662 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:34.324678 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:34.324684 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:34.324843 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:34.325249 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:34.325256 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:34.325259 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:34.325836 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:34.326158 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:34.326624 (+ 466us) server_negotiation.cc:299] Negotiation successful
0928 14:58:34.327098 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":867,"thread_start_us":586,"threads_started":1}
I20260928 14:58:34.328410 10892 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3b972e6f1bd40e38fc47fa4ec552e2a [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:34.331630 10892 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3b972e6f1bd40e38fc47fa4ec552e2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3b972e6f1bd40e38fc47fa4ec552e2a" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32879 } }
I20260928 14:58:34.332199 10892 leader_election.cc:302] T 00000000000000000000000000000000 P a3b972e6f1bd40e38fc47fa4ec552e2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3b972e6f1bd40e38fc47fa4ec552e2a; no voters:
I20260928 14:58:34.332718 10892 leader_election.cc:288] T 00000000000000000000000000000000 P a3b972e6f1bd40e38fc47fa4ec552e2a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:58:34.332881 10897 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3b972e6f1bd40e38fc47fa4ec552e2a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:58:34.333689 10897 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3b972e6f1bd40e38fc47fa4ec552e2a [term 1 LEADER]: Becoming Leader. State: Replica: a3b972e6f1bd40e38fc47fa4ec552e2a, State: Running, Role: LEADER
I20260928 14:58:34.334283 10897 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3b972e6f1bd40e38fc47fa4ec552e2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3b972e6f1bd40e38fc47fa4ec552e2a" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32879 } }
I20260928 14:58:34.334899 10892 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3b972e6f1bd40e38fc47fa4ec552e2a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:58:34.336834 10898 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3b972e6f1bd40e38fc47fa4ec552e2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3b972e6f1bd40e38fc47fa4ec552e2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3b972e6f1bd40e38fc47fa4ec552e2a" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32879 } } }
I20260928 14:58:34.337179 10898 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3b972e6f1bd40e38fc47fa4ec552e2a [sys.catalog]: This master's current role is: LEADER
I20260928 14:58:34.338686 10899 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3b972e6f1bd40e38fc47fa4ec552e2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3b972e6f1bd40e38fc47fa4ec552e2a. Latest consensus state: current_term: 1 leader_uuid: "a3b972e6f1bd40e38fc47fa4ec552e2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3b972e6f1bd40e38fc47fa4ec552e2a" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 32879 } } }
I20260928 14:58:34.339056 10899 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3b972e6f1bd40e38fc47fa4ec552e2a [sys.catalog]: This master's current role is: LEADER
I20260928 14:58:34.340483 10902 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:58:34.347613 10902 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:58:34.350564 10892 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 14:58:34.368942 10902 catalog_manager.cc:1347] Generated new cluster ID: c0e710d62754497aaf3b486d0596f596
I20260928 14:58:34.369078 10902 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:58:34.418066 10902 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:58:34.419795 10902 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:58:34.430028 10902 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3b972e6f1bd40e38fc47fa4ec552e2a: Generated new TSK 0
I20260928 14:58:34.430972 10902 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.40.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab.
Entry for principal kudu/127.31.40.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab.
I20260928 14:58:35.260483 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.40.20:35655/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.1:0
--local_ip_for_outbound_sockets=127.31.40.1
--webserver_interface=127.31.40.1
--webserver_port=0
--tserver_master_addrs=127.31.40.62:32879
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab
--principal=kudu/127.31.40.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:41653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 14:58:35 dist-test-slave-j968 krb5kdc[10380](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790607515, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 14:58:35.706240 10937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:35.706774 10937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:35.706967 10937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:35.718806 10937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:35.718963 10937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:35.719223 10937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.1
I20260928 14:58:35.735672 10937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:41653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.31.40.20:35655/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab
--principal=kudu/127.31.40.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.40.1
--webserver_port=0
--tserver_master_addrs=127.31.40.62:32879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.1
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:58:35.737757 10937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:35.739545 10937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:35.757954 10948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:35.758117 10950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:35.758839 10937 server_base.cc:1034] running on GCE node
W20260928 14:58:35.760129 10947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:35.761111 10937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:35.762534 10937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:35.763906 10937 hybrid_clock.cc:648] HybridClock initialized: now 1790607515763756 us; error 98 us; skew 500 ppm
I20260928 14:58:35.764338 10937 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:58:35 dist-test-slave-j968 krb5kdc[10380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607515, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:35.769073 10937 init.cc:377] Logged in from keytab as kudu/127.31.40.1@KRBTEST.COM (short username kudu)
I20260928 14:58:35.771265 10937 webserver.cc:460] Webserver started at http://127.31.40.1:43075/ using document root <none> and password file <none>
I20260928 14:58:35.772300 10937 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:35.772467 10937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:35.772887 10937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 14:58:35 dist-test-slave-j968 krb5kdc[10380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607515, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.1@KRBTEST.COM for HTTP/127.31.40.20@KRBTEST.COM
I20260928 14:58:35.797533 10937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ts-0/data/instance:
uuid: "beabd747c43e4c8d8c6d23b1a588befc"
format_stamp: "Formatted at 2026-09-28 14:58:35 on dist-test-slave-j968"
server_key: "735d1778e84d0b3696b43b61bdbdfc5e"
server_key_iv: "b46d0042d5b0476258589a207e0dbd54"
server_key_version: "kuduclusterkey@0"
I20260928 14:58:35.799666 10937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ts-0/wal/instance:
uuid: "beabd747c43e4c8d8c6d23b1a588befc"
format_stamp: "Formatted at 2026-09-28 14:58:35 on dist-test-slave-j968"
server_key: "735d1778e84d0b3696b43b61bdbdfc5e"
server_key_iv: "b46d0042d5b0476258589a207e0dbd54"
server_key_version: "kuduclusterkey@0"
I20260928 14:58:35.820112 10937 fs_manager.cc:691] Time spent creating directory manager: real 0.020s user 0.009s sys 0.008s
I20260928 14:58:35.852766 10957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:35.855125 10937 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.011s sys 0.004s
I20260928 14:58:35.855427 10937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ts-0/wal
uuid: "beabd747c43e4c8d8c6d23b1a588befc"
format_stamp: "Formatted at 2026-09-28 14:58:35 on dist-test-slave-j968"
server_key: "735d1778e84d0b3696b43b61bdbdfc5e"
server_key_iv: "b46d0042d5b0476258589a207e0dbd54"
server_key_version: "kuduclusterkey@0"
I20260928 14:58:35.855885 10937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:35.927366 10937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:35.933593 10937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:35.933998 10937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:35.937062 10937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:58:35.938860 10937 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:58:35.938975 10937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:35.939160 10937 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:58:35.939247 10937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:36.023537 10937 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.1:38393
I20260928 14:58:36.023587 11069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.1:38393 every 8 connection(s)
I20260928 14:58:36.025321 10937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
I20260928 14:58:36.029801 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 10937
I20260928 14:58:36.030057 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ts-0/wal/instance
I20260928 14:58:36.031087 31904 external_mini_cluster.cc:1428] Setting key 59773d52c267211cbc9e114b9797d674
Sep 28 14:58:36 dist-test-slave-j968 krb5kdc[10380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607515, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.1@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.40.2@KRBTEST.COM; defaulting to no policy
I20260928 14:58:36.099725 11073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:36.035350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:35781 (local address 127.31.40.62:32879)
0928 14:58:36.062451 (+ 27101us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:36.062458 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:36.067923 (+ 5465us) server_negotiation.cc:407] Connection header received
0928 14:58:36.068008 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:36.068014 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:36.068117 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:36.068224 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:36.075035 (+ 6811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:36.076259 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:36.078886 (+ 2627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:36.079591 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:36.079735 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:36.091052 (+ 11317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:36.091061 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:36.091066 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:36.091115 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:36.095476 (+ 4361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:36.096607 (+ 1131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:36.096614 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:36.096618 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:36.096769 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:36.097365 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:36.097372 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:36.097376 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:36.097787 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:36.097957 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:36.098985 (+ 1028us) server_negotiation.cc:299] Negotiation successful
0928 14:58:36.099298 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26913,"thread_start_us":27184,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.2@KRBTEST.COM" created.
I20260928 14:58:36.100999 11072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:36.034656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:32879 (local address 127.31.40.1:35781)
0928 14:58:36.063735 (+ 29079us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:36.063917 (+ 182us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:36.067372 (+ 3455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:36.072332 (+ 4960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:36.072346 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:36.073411 (+ 1065us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:36.074738 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:36.074774 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:36.076531 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:36.076537 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:36.078657 (+ 2120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:36.078676 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:36.085596 (+ 6920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:36.085607 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:36.085997 (+ 390us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:36.086923 (+ 926us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:36.086959 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:36.090576 (+ 3617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:36.095739 (+ 5163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:36.095750 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:36.095774 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:36.096446 (+ 672us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:36.096912 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:36.096917 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:36.096920 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:36.097188 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:36.097988 (+ 800us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:36.097998 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:36.098490 (+ 492us) client_negotiation.cc:769] Sending connection context
0928 14:58:36.098834 (+ 344us) client_negotiation.cc:240] Negotiation successful
0928 14:58:36.099435 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28563,"thread_start_us":35290,"threads_started":1}
I20260928 14:58:36.103338 11070 heartbeater.cc:347] Connected to a master server at 127.31.40.62:32879
I20260928 14:58:36.103686 11070 heartbeater.cc:464] Registering TS with master...
I20260928 14:58:36.104667 11070 heartbeater.cc:511] Master 127.31.40.62:32879 requested a full tablet report, sending...
I20260928 14:58:36.108358 10857 ts_manager.cc:194] Registered new tserver with Master: beabd747c43e4c8d8c6d23b1a588befc (127.31.40.1:38393)
I20260928 14:58:36.110598 10857 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.1@KRBTEST.COM'} at 127.31.40.1:35781
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab.
Entry for principal kudu/127.31.40.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab.
W20260928 14:58:36.138907 31904 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.31.40.2: real 0.104s user 0.001s sys 0.018s
I20260928 14:58:36.139909 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.40.20:35655/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.2:0
--local_ip_for_outbound_sockets=127.31.40.2
--webserver_interface=127.31.40.2
--webserver_port=0
--tserver_master_addrs=127.31.40.62:32879
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab
--principal=kudu/127.31.40.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:41653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 14:58:36 dist-test-slave-j968 krb5kdc[10380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607515, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.62@KRBTEST.COM for HTTP/127.31.40.20@KRBTEST.COM
Sep 28 14:58:36 dist-test-slave-j968 krb5kdc[10380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607515, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.62@KRBTEST.COM for HTTP/127.31.40.20@KRBTEST.COM
W20260928 14:58:36.713424 11076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:36.713855 11076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:36.714026 11076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:36.725786 11076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:36.725997 11076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:36.726198 11076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.2
I20260928 14:58:36.742494 11076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:41653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.31.40.20:35655/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab
--principal=kudu/127.31.40.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.40.2
--webserver_port=0
--tserver_master_addrs=127.31.40.62:32879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.2
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:58:36.744544 11076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:36.746491 11076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:36.767349 11086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:36.769407 11088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:36.770457 11076 server_base.cc:1034] running on GCE node
W20260928 14:58:36.772192 11085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:36.773108 11076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:36.774296 11076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:36.775669 11076 hybrid_clock.cc:648] HybridClock initialized: now 1790607516775554 us; error 107 us; skew 500 ppm
I20260928 14:58:36.776209 11076 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:58:36 dist-test-slave-j968 krb5kdc[10380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607516, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:36.780720 11076 init.cc:377] Logged in from keytab as kudu/127.31.40.2@KRBTEST.COM (short username kudu)
I20260928 14:58:36.782685 11076 webserver.cc:460] Webserver started at http://127.31.40.2:37811/ using document root <none> and password file <none>
I20260928 14:58:36.783588 11076 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:36.783818 11076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:36.784190 11076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 14:58:36 dist-test-slave-j968 krb5kdc[10380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607516, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.2@KRBTEST.COM for HTTP/127.31.40.20@KRBTEST.COM
I20260928 14:58:36.798254 11076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ts-1/data/instance:
uuid: "3216c8ce2e1e4a06b9f2eb16e42838f1"
format_stamp: "Formatted at 2026-09-28 14:58:36 on dist-test-slave-j968"
server_key: "503ef2c810b791c2ef933c193ea2e7ea"
server_key_iv: "6e3016d7cf013056c69374f4b12e3859"
server_key_version: "kuduclusterkey@0"
I20260928 14:58:36.799170 11076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ts-1/wal/instance:
uuid: "3216c8ce2e1e4a06b9f2eb16e42838f1"
format_stamp: "Formatted at 2026-09-28 14:58:36 on dist-test-slave-j968"
server_key: "503ef2c810b791c2ef933c193ea2e7ea"
server_key_iv: "6e3016d7cf013056c69374f4b12e3859"
server_key_version: "kuduclusterkey@0"
I20260928 14:58:36.805079 11076 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260928 14:58:36.819968 11100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:36.821684 11076 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 14:58:36.821897 11076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ts-1/wal
uuid: "3216c8ce2e1e4a06b9f2eb16e42838f1"
format_stamp: "Formatted at 2026-09-28 14:58:36 on dist-test-slave-j968"
server_key: "503ef2c810b791c2ef933c193ea2e7ea"
server_key_iv: "6e3016d7cf013056c69374f4b12e3859"
server_key_version: "kuduclusterkey@0"
I20260928 14:58:36.822153 11076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:36.838682 11076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:36.843335 11076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:36.843746 11076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:36.844867 11076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:58:36.846560 11076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:58:36.846669 11076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:36.846822 11076 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:58:36.846882 11076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:36.894755 11076 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.2:36761
I20260928 14:58:36.894780 11212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.2:36761 every 8 connection(s)
I20260928 14:58:36.896154 11076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
I20260928 14:58:36.900549 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 11076
I20260928 14:58:36.900727 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ts-1/wal/instance
I20260928 14:58:36.901295 31904 external_mini_cluster.cc:1428] Setting key 7a14d8e23a9dbbe8c5b916331488cdc0
Sep 28 14:58:36 dist-test-slave-j968 krb5kdc[10380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607516, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.2@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:58:36.922381 11215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:36.899222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:42407 (local address 127.31.40.62:32879)
0928 14:58:36.899719 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:36.899726 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:36.900932 (+ 1206us) server_negotiation.cc:407] Connection header received
0928 14:58:36.903689 (+ 2757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:36.903696 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:36.903777 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:36.903930 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:36.905921 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:36.907158 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:36.909640 (+ 2482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:36.910346 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:36.910493 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:36.915812 (+ 5319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:36.915818 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:36.915821 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:36.915919 (+ 98us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:36.918620 (+ 2701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:36.919601 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:36.919606 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:36.919608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:36.919738 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:36.920360 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:36.920365 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:36.920368 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:36.920666 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:36.920854 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:36.921699 (+ 845us) server_negotiation.cc:299] Negotiation successful
0928 14:58:36.921980 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":230,"threads_started":1}
I20260928 14:58:36.923640 11216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:36.899777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:32879 (local address 127.31.40.2:42407)
0928 14:58:36.900764 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:36.900815 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:36.903334 (+ 2519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:36.904223 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:36.904235 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:36.904864 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:36.905706 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:36.905731 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:36.907345 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:36.907351 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:36.909448 (+ 2097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:36.909465 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:36.910588 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:36.910595 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:36.910848 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:36.911610 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:36.911646 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:36.915000 (+ 3354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:36.918793 (+ 3793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:36.918804 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:36.918825 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:36.919450 (+ 625us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:36.919860 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:36.919901 (+ 41us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:36.919905 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:36.920223 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:36.920812 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:36.920822 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:36.921254 (+ 432us) client_negotiation.cc:769] Sending connection context
0928 14:58:36.921567 (+ 313us) client_negotiation.cc:240] Negotiation successful
0928 14:58:36.922183 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"thread_start_us":379,"threads_started":1}
I20260928 14:58:36.925426 11213 heartbeater.cc:347] Connected to a master server at 127.31.40.62:32879
I20260928 14:58:36.925779 11213 heartbeater.cc:464] Registering TS with master...
I20260928 14:58:36.926942 11213 heartbeater.cc:511] Master 127.31.40.62:32879 requested a full tablet report, sending...
I20260928 14:58:36.928646 10857 ts_manager.cc:194] Registered new tserver with Master: 3216c8ce2e1e4a06b9f2eb16e42838f1 (127.31.40.2:36761)
I20260928 14:58:36.929500 10857 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.2@KRBTEST.COM'} at 127.31.40.2:42407
WARNING: no policy specified for kudu/127.31.40.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab.
Entry for principal kudu/127.31.40.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab.
I20260928 14:58:36.968565 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.40.20:35655/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.3:0
--local_ip_for_outbound_sockets=127.31.40.3
--webserver_interface=127.31.40.3
--webserver_port=0
--tserver_master_addrs=127.31.40.62:32879
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab
--principal=kudu/127.31.40.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:41653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:58:37.115031 11070 heartbeater.cc:503] Master 127.31.40.62:32879 was elected leader, sending a full tablet report...
W20260928 14:58:37.347898 11219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:37.348206 11219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:37.348304 11219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:37.356181 11219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:37.356297 11219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:37.356418 11219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.3
I20260928 14:58:37.367087 11219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:41653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.31.40.20:35655/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab
--principal=kudu/127.31.40.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.40.3
--webserver_port=0
--tserver_master_addrs=127.31.40.62:32879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.3
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:58:37.368530 11219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:37.369813 11219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:37.385365 11227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:37.385466 11224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:37.385470 11225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:37.386193 11219 server_base.cc:1034] running on GCE node
I20260928 14:58:37.386827 11219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:37.387856 11219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:37.389110 11219 hybrid_clock.cc:648] HybridClock initialized: now 1790607517389042 us; error 61 us; skew 500 ppm
I20260928 14:58:37.389436 11219 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:58:37 dist-test-slave-j968 krb5kdc[10380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607517, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:37.393344 11219 init.cc:377] Logged in from keytab as kudu/127.31.40.3@KRBTEST.COM (short username kudu)
I20260928 14:58:37.395167 11219 webserver.cc:460] Webserver started at http://127.31.40.3:40465/ using document root <none> and password file <none>
I20260928 14:58:37.395993 11219 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:37.396178 11219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:37.396522 11219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 14:58:37 dist-test-slave-j968 krb5kdc[10380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607517, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.3@KRBTEST.COM for HTTP/127.31.40.20@KRBTEST.COM
I20260928 14:58:37.411923 11219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ts-2/data/instance:
uuid: "0f61915fb9ad4709874b6b96a0023381"
format_stamp: "Formatted at 2026-09-28 14:58:37 on dist-test-slave-j968"
server_key: "2f196a035413e6b68d0d2261ea996512"
server_key_iv: "84d316c4b393acf86c8b224d4058a79e"
server_key_version: "kuduclusterkey@0"
I20260928 14:58:37.413095 11219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ts-2/wal/instance:
uuid: "0f61915fb9ad4709874b6b96a0023381"
format_stamp: "Formatted at 2026-09-28 14:58:37 on dist-test-slave-j968"
server_key: "2f196a035413e6b68d0d2261ea996512"
server_key_iv: "84d316c4b393acf86c8b224d4058a79e"
server_key_version: "kuduclusterkey@0"
I20260928 14:58:37.420712 11219 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.003s
I20260928 14:58:37.437795 11234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:37.439832 11219 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 14:58:37.440078 11219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ts-2/wal
uuid: "0f61915fb9ad4709874b6b96a0023381"
format_stamp: "Formatted at 2026-09-28 14:58:37 on dist-test-slave-j968"
server_key: "2f196a035413e6b68d0d2261ea996512"
server_key_iv: "84d316c4b393acf86c8b224d4058a79e"
server_key_version: "kuduclusterkey@0"
I20260928 14:58:37.440366 11219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:37.469203 11219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:37.474123 11219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:37.474469 11219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:37.475543 11219 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:58:37.477079 11219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:58:37.477238 11219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:37.477528 11219 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:58:37.477595 11219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:37.518957 11219 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.3:40889
I20260928 14:58:37.519240 11346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.3:40889 every 8 connection(s)
I20260928 14:58:37.520458 11219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
I20260928 14:58:37.528599 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 11219
I20260928 14:58:37.528782 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ts-2/wal/instance
I20260928 14:58:37.529354 31904 external_mini_cluster.cc:1428] Setting key 053340297e39cc9ca727084bc0b34f38
Sep 28 14:58:37 dist-test-slave-j968 krb5kdc[10380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607517, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.3@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:58:37.547051 11349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:37.523386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:56375 (local address 127.31.40.62:32879)
0928 14:58:37.524051 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:37.524060 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:37.526685 (+ 2625us) server_negotiation.cc:407] Connection header received
0928 14:58:37.529204 (+ 2519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:37.529214 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:37.529345 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:37.529480 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:37.532000 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:37.533203 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:37.535140 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:37.535677 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:37.535951 (+ 274us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:37.539649 (+ 3698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:37.539654 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:37.539658 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:37.539695 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:37.543071 (+ 3376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:37.543904 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:37.543913 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:37.543916 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:37.544174 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:37.544953 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:37.544957 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:37.544960 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:37.545366 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:37.545537 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:37.546531 (+ 994us) server_negotiation.cc:299] Negotiation successful
0928 14:58:37.546731 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":250,"threads_started":1}
I20260928 14:58:37.548352 11350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:37.523996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:32879 (local address 127.31.40.3:56375)
0928 14:58:37.526482 (+ 2486us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:37.526538 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:37.528818 (+ 2280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:37.529855 (+ 1037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:37.529868 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:37.530471 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:37.531660 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:37.531684 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:37.533432 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:37.533440 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:37.534976 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:37.534990 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:37.536044 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:37.536053 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:37.536285 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:37.537097 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:37.537128 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:37.539409 (+ 2281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:37.543250 (+ 3841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:37.543258 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:37.543273 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:37.543748 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:37.544308 (+ 560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:37.544312 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:37.544315 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:37.544758 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:37.545546 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:37.545555 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:37.546019 (+ 464us) client_negotiation.cc:769] Sending connection context
0928 14:58:37.546445 (+ 426us) client_negotiation.cc:240] Negotiation successful
0928 14:58:37.547041 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2201,"thread_start_us":2004,"threads_started":1}
I20260928 14:58:37.550076 11347 heartbeater.cc:347] Connected to a master server at 127.31.40.62:32879
I20260928 14:58:37.550414 11347 heartbeater.cc:464] Registering TS with master...
I20260928 14:58:37.551246 11347 heartbeater.cc:511] Master 127.31.40.62:32879 requested a full tablet report, sending...
I20260928 14:58:37.553663 10857 ts_manager.cc:194] Registered new tserver with Master: 0f61915fb9ad4709874b6b96a0023381 (127.31.40.3:40889)
I20260928 14:58:37.554850 10857 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.3@KRBTEST.COM'} at 127.31.40.3:56375
I20260928 14:58:37.559396 31904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:58:37.579511 11349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:37.565011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42368 (local address 127.31.40.62:32879)
0928 14:58:37.565190 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:37.565195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:37.565740 (+ 545us) server_negotiation.cc:407] Connection header received
0928 14:58:37.565954 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:37.565964 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:37.566045 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:37.566156 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:37.567886 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:37.568684 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:37.570136 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:37.570597 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:37.570767 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:37.573326 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:37.573331 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:37.573334 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:37.573371 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:37.576566 (+ 3195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:37.577406 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:37.577412 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:37.577416 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:37.577551 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:37.577938 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:37.577945 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:37.577949 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:37.578238 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:37.578504 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:37.578922 (+ 418us) server_negotiation.cc:299] Negotiation successful
0928 14:58:37.579197 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 14:58:37.584981 10857 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42368:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:58:37.595424 10857 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:58:37.621605 11361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:37.610037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48880 (local address 127.31.40.3:40889)
0928 14:58:37.610971 (+ 934us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:37.610980 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:37.612572 (+ 1592us) server_negotiation.cc:407] Connection header received
0928 14:58:37.612874 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:37.612886 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:37.613179 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:37.613395 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:58:37.614221 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:37.615889 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:37.618467 (+ 2578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:37.620349 (+ 1882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:37.620770 (+ 421us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:37.621041 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:37.621109 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 14:58:37.621270 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":682,"thread_start_us":342,"threads_started":1}
I20260928 14:58:37.621644 11360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:37.609854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.3:40889 (local address 127.0.0.1:48880)
0928 14:58:37.612428 (+ 2574us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:37.612459 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:37.612667 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:37.613424 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:37.613435 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:37.613473 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:58:37.613940 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:37.613959 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:37.616047 (+ 2088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:37.616052 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:37.618314 (+ 2262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:37.618325 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:37.620587 (+ 2262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:37.620593 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:37.620826 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:37.620876 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 14:58:37.620996 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 14:58:37.621197 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2375,"mutex_wait_us":17,"thread_start_us":2161,"threads_started":1}
I20260928 14:58:37.622820 11362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:37.610615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.2:36761 (local address 127.0.0.1:40708)
0928 14:58:37.613246 (+ 2631us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:37.613263 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:37.613381 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:37.615195 (+ 1814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:37.615203 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:37.615235 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:58:37.615674 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:37.615684 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:37.617755 (+ 2071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:37.617761 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:37.619847 (+ 2086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:37.619859 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:37.622009 (+ 2150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:37.622015 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:37.622190 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:37.622226 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 14:58:37.622314 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 14:58:37.622444 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2504,"thread_start_us":2358,"threads_started":1}
I20260928 14:58:37.622886 11363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:37.611050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40708 (local address 127.31.40.2:36761)
0928 14:58:37.611972 (+ 922us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:37.611981 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:37.614642 (+ 2661us) server_negotiation.cc:407] Connection header received
0928 14:58:37.614748 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:37.614755 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:37.615001 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:37.615181 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:58:37.615840 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:37.617633 (+ 1793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:37.620076 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:37.621511 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:37.621626 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:37.621845 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:37.622391 (+ 546us) server_negotiation.cc:299] Negotiation successful
0928 14:58:37.622552 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":738,"thread_start_us":613,"threads_started":1}
I20260928 14:58:37.624572 11364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:37.613732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:38393 (local address 127.0.0.1:51726)
0928 14:58:37.614415 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:37.614437 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:37.614569 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:37.616371 (+ 1802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:37.616376 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:37.616400 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:58:37.616863 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:37.616880 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:37.618973 (+ 2093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:37.618978 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:37.621871 (+ 2893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:37.621887 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:37.623784 (+ 1897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:37.623790 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:37.624007 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:37.624044 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 14:58:37.624145 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 14:58:37.624262 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"mutex_wait_us":24,"thread_start_us":365,"threads_started":1}
I20260928 14:58:37.626549 11282 tablet_service.cc:1467] Processing CreateTablet for tablet c06137b65d424f5fb14cfdbaa1819e8d (DEFAULT_TABLE table=test-table [id=8ece6eba5b4f4bc19bd01c01fc4ad697]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:58:37.626765 11148 tablet_service.cc:1467] Processing CreateTablet for tablet c06137b65d424f5fb14cfdbaa1819e8d (DEFAULT_TABLE table=test-table [id=8ece6eba5b4f4bc19bd01c01fc4ad697]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:58:37.628196 11148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c06137b65d424f5fb14cfdbaa1819e8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:37.628196 11282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c06137b65d424f5fb14cfdbaa1819e8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:37.628598 11365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:37.613891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51726 (local address 127.31.40.1:38393)
0928 14:58:37.615822 (+ 1931us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:37.615830 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:37.615857 (+ 27us) server_negotiation.cc:407] Connection header received
0928 14:58:37.615981 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:37.615988 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:37.616180 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:37.616382 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:58:37.617193 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:37.618774 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:37.622059 (+ 3285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:37.623574 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:37.623705 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:37.623955 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:37.628058 (+ 4103us) server_negotiation.cc:299] Negotiation successful
0928 14:58:37.628223 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1766,"thread_start_us":1603,"threads_started":1}
I20260928 14:58:37.632405 11005 tablet_service.cc:1467] Processing CreateTablet for tablet c06137b65d424f5fb14cfdbaa1819e8d (DEFAULT_TABLE table=test-table [id=8ece6eba5b4f4bc19bd01c01fc4ad697]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:58:37.633982 11005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c06137b65d424f5fb14cfdbaa1819e8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:37.645759 11366 tablet_bootstrap.cc:492] T c06137b65d424f5fb14cfdbaa1819e8d P 0f61915fb9ad4709874b6b96a0023381: Bootstrap starting.
I20260928 14:58:37.646910 11367 tablet_bootstrap.cc:492] T c06137b65d424f5fb14cfdbaa1819e8d P 3216c8ce2e1e4a06b9f2eb16e42838f1: Bootstrap starting.
I20260928 14:58:37.650066 11368 tablet_bootstrap.cc:492] T c06137b65d424f5fb14cfdbaa1819e8d P beabd747c43e4c8d8c6d23b1a588befc: Bootstrap starting.
I20260928 14:58:37.651337 11366 tablet_bootstrap.cc:654] T c06137b65d424f5fb14cfdbaa1819e8d P 0f61915fb9ad4709874b6b96a0023381: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:37.651340 11367 tablet_bootstrap.cc:654] T c06137b65d424f5fb14cfdbaa1819e8d P 3216c8ce2e1e4a06b9f2eb16e42838f1: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:37.652855 11367 log.cc:824] T c06137b65d424f5fb14cfdbaa1819e8d P 3216c8ce2e1e4a06b9f2eb16e42838f1: Log is configured to *not* fsync() on all Append() calls
I20260928 14:58:37.652964 11366 log.cc:824] T c06137b65d424f5fb14cfdbaa1819e8d P 0f61915fb9ad4709874b6b96a0023381: Log is configured to *not* fsync() on all Append() calls
I20260928 14:58:37.653446 11368 tablet_bootstrap.cc:654] T c06137b65d424f5fb14cfdbaa1819e8d P beabd747c43e4c8d8c6d23b1a588befc: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:37.654558 11368 log.cc:824] T c06137b65d424f5fb14cfdbaa1819e8d P beabd747c43e4c8d8c6d23b1a588befc: Log is configured to *not* fsync() on all Append() calls
I20260928 14:58:37.657946 11368 tablet_bootstrap.cc:492] T c06137b65d424f5fb14cfdbaa1819e8d P beabd747c43e4c8d8c6d23b1a588befc: No bootstrap required, opened a new log
I20260928 14:58:37.658244 11366 tablet_bootstrap.cc:492] T c06137b65d424f5fb14cfdbaa1819e8d P 0f61915fb9ad4709874b6b96a0023381: No bootstrap required, opened a new log
I20260928 14:58:37.658253 11368 ts_tablet_manager.cc:1397] T c06137b65d424f5fb14cfdbaa1819e8d P beabd747c43e4c8d8c6d23b1a588befc: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260928 14:58:37.658543 11366 ts_tablet_manager.cc:1397] T c06137b65d424f5fb14cfdbaa1819e8d P 0f61915fb9ad4709874b6b96a0023381: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.005s
I20260928 14:58:37.658762 11367 tablet_bootstrap.cc:492] T c06137b65d424f5fb14cfdbaa1819e8d P 3216c8ce2e1e4a06b9f2eb16e42838f1: No bootstrap required, opened a new log
I20260928 14:58:37.659236 11367 ts_tablet_manager.cc:1397] T c06137b65d424f5fb14cfdbaa1819e8d P 3216c8ce2e1e4a06b9f2eb16e42838f1: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260928 14:58:37.667639 11368 raft_consensus.cc:348] T c06137b65d424f5fb14cfdbaa1819e8d P beabd747c43e4c8d8c6d23b1a588befc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f61915fb9ad4709874b6b96a0023381" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 40889 } } peers { permanent_uuid: "beabd747c43e4c8d8c6d23b1a588befc" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 38393 } } peers { permanent_uuid: "3216c8ce2e1e4a06b9f2eb16e42838f1" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 36761 } }
I20260928 14:58:37.668207 11368 raft_consensus.cc:374] T c06137b65d424f5fb14cfdbaa1819e8d P beabd747c43e4c8d8c6d23b1a588befc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:37.668339 11368 raft_consensus.cc:729] T c06137b65d424f5fb14cfdbaa1819e8d P beabd747c43e4c8d8c6d23b1a588befc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beabd747c43e4c8d8c6d23b1a588befc, State: Initialized, Role: FOLLOWER
I20260928 14:58:37.669018 11368 consensus_queue.cc:260] T c06137b65d424f5fb14cfdbaa1819e8d P beabd747c43e4c8d8c6d23b1a588befc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f61915fb9ad4709874b6b96a0023381" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 40889 } } peers { permanent_uuid: "beabd747c43e4c8d8c6d23b1a588befc" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 38393 } } peers { permanent_uuid: "3216c8ce2e1e4a06b9f2eb16e42838f1" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 36761 } }
I20260928 14:58:37.669979 11366 raft_consensus.cc:348] T c06137b65d424f5fb14cfdbaa1819e8d P 0f61915fb9ad4709874b6b96a0023381 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f61915fb9ad4709874b6b96a0023381" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 40889 } } peers { permanent_uuid: "beabd747c43e4c8d8c6d23b1a588befc" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 38393 } } peers { permanent_uuid: "3216c8ce2e1e4a06b9f2eb16e42838f1" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 36761 } }
I20260928 14:58:37.670564 11366 raft_consensus.cc:374] T c06137b65d424f5fb14cfdbaa1819e8d P 0f61915fb9ad4709874b6b96a0023381 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:37.670653 11366 raft_consensus.cc:729] T c06137b65d424f5fb14cfdbaa1819e8d P 0f61915fb9ad4709874b6b96a0023381 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f61915fb9ad4709874b6b96a0023381, State: Initialized, Role: FOLLOWER
I20260928 14:58:37.670431 11367 raft_consensus.cc:348] T c06137b65d424f5fb14cfdbaa1819e8d P 3216c8ce2e1e4a06b9f2eb16e42838f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f61915fb9ad4709874b6b96a0023381" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 40889 } } peers { permanent_uuid: "beabd747c43e4c8d8c6d23b1a588befc" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 38393 } } peers { permanent_uuid: "3216c8ce2e1e4a06b9f2eb16e42838f1" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 36761 } }
I20260928 14:58:37.670864 11367 raft_consensus.cc:374] T c06137b65d424f5fb14cfdbaa1819e8d P 3216c8ce2e1e4a06b9f2eb16e42838f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:37.670931 11368 ts_tablet_manager.cc:1428] T c06137b65d424f5fb14cfdbaa1819e8d P beabd747c43e4c8d8c6d23b1a588befc: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260928 14:58:37.670958 11367 raft_consensus.cc:729] T c06137b65d424f5fb14cfdbaa1819e8d P 3216c8ce2e1e4a06b9f2eb16e42838f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3216c8ce2e1e4a06b9f2eb16e42838f1, State: Initialized, Role: FOLLOWER
I20260928 14:58:37.671511 11366 consensus_queue.cc:260] T c06137b65d424f5fb14cfdbaa1819e8d P 0f61915fb9ad4709874b6b96a0023381 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f61915fb9ad4709874b6b96a0023381" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 40889 } } peers { permanent_uuid: "beabd747c43e4c8d8c6d23b1a588befc" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 38393 } } peers { permanent_uuid: "3216c8ce2e1e4a06b9f2eb16e42838f1" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 36761 } }
I20260928 14:58:37.671511 11367 consensus_queue.cc:260] T c06137b65d424f5fb14cfdbaa1819e8d P 3216c8ce2e1e4a06b9f2eb16e42838f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f61915fb9ad4709874b6b96a0023381" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 40889 } } peers { permanent_uuid: "beabd747c43e4c8d8c6d23b1a588befc" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 38393 } } peers { permanent_uuid: "3216c8ce2e1e4a06b9f2eb16e42838f1" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 36761 } }
I20260928 14:58:37.672686 11347 heartbeater.cc:503] Master 127.31.40.62:32879 was elected leader, sending a full tablet report...
I20260928 14:58:37.673586 11366 ts_tablet_manager.cc:1428] T c06137b65d424f5fb14cfdbaa1819e8d P 0f61915fb9ad4709874b6b96a0023381: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260928 14:58:37.674654 11213 heartbeater.cc:503] Master 127.31.40.62:32879 was elected leader, sending a full tablet report...
I20260928 14:58:37.675249 11367 ts_tablet_manager.cc:1428] T c06137b65d424f5fb14cfdbaa1819e8d P 3216c8ce2e1e4a06b9f2eb16e42838f1: Time spent starting tablet: real 0.016s user 0.013s sys 0.005s
W20260928 14:58:37.772294 11348 tablet.cc:2369] T c06137b65d424f5fb14cfdbaa1819e8d P 0f61915fb9ad4709874b6b96a0023381: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:58:37.821275 11374 raft_consensus.cc:482] T c06137b65d424f5fb14cfdbaa1819e8d P 3216c8ce2e1e4a06b9f2eb16e42838f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:58:37.821630 11374 raft_consensus.cc:504] T c06137b65d424f5fb14cfdbaa1819e8d P 3216c8ce2e1e4a06b9f2eb16e42838f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f61915fb9ad4709874b6b96a0023381" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 40889 } } peers { permanent_uuid: "beabd747c43e4c8d8c6d23b1a588befc" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 38393 } } peers { permanent_uuid: "3216c8ce2e1e4a06b9f2eb16e42838f1" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 36761 } }
I20260928 14:58:37.823642 11374 leader_election.cc:288] T c06137b65d424f5fb14cfdbaa1819e8d P 3216c8ce2e1e4a06b9f2eb16e42838f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f61915fb9ad4709874b6b96a0023381 (127.31.40.3:40889), beabd747c43e4c8d8c6d23b1a588befc (127.31.40.1:38393)
W20260928 14:58:37.823678 11071 tablet.cc:2369] T c06137b65d424f5fb14cfdbaa1819e8d P beabd747c43e4c8d8c6d23b1a588befc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:58:37.831661 11375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:37.824026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.3:40889 (local address 127.31.40.2:37407)
0928 14:58:37.824715 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:37.824755 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:37.825011 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:37.825492 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:37.825498 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:37.825538 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:58:37.826013 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:37.826026 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:37.827592 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:37.827597 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:37.829304 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:37.829316 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:37.830869 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:37.830876 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:37.831057 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:37.831107 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 14:58:37.831200 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 14:58:37.831351 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":356,"threads_started":1}
I20260928 14:58:37.831694 11361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:37.824239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:37407 (local address 127.31.40.3:40889)
0928 14:58:37.824510 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:37.824519 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:37.824878 (+ 359us) server_negotiation.cc:407] Connection header received
0928 14:58:37.825182 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:37.825190 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:37.825294 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:37.825415 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:58:37.826249 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:37.827423 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:37.829481 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:37.830644 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:37.830753 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:37.830846 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:37.831259 (+ 413us) server_negotiation.cc:299] Negotiation successful
0928 14:58:37.831420 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260928 14:58:37.832290 11365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:37.825126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:50331 (local address 127.31.40.1:38393)
0928 14:58:37.825384 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:37.825390 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:37.825777 (+ 387us) server_negotiation.cc:407] Connection header received
0928 14:58:37.825990 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:37.825997 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:37.826086 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:37.826187 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:58:37.827014 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:37.827970 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:37.830087 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:37.831025 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:37.831167 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:37.831270 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:37.831779 (+ 509us) server_negotiation.cc:299] Negotiation successful
0928 14:58:37.832119 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260928 14:58:37.832489 11376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:37.824832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:38393 (local address 127.31.40.2:50331)
0928 14:58:37.825640 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:37.825655 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:37.825827 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:37.826251 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:37.826256 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:37.826286 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:58:37.826731 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:37.826744 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:37.828114 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:37.828122 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:37.829908 (+ 1786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:37.829925 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:37.831284 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:37.831291 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:37.831558 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:37.831600 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 14:58:37.831731 (+ 131us) client_negotiation.cc:240] Negotiation successful
0928 14:58:37.832101 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":464,"threads_started":1}
I20260928 14:58:37.832989 11302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c06137b65d424f5fb14cfdbaa1819e8d" candidate_uuid: "3216c8ce2e1e4a06b9f2eb16e42838f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f61915fb9ad4709874b6b96a0023381" is_pre_election: true
I20260928 14:58:37.833528 11302 raft_consensus.cc:2454] T c06137b65d424f5fb14cfdbaa1819e8d P 0f61915fb9ad4709874b6b96a0023381 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3216c8ce2e1e4a06b9f2eb16e42838f1 in term 0.
I20260928 14:58:37.833500 11025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c06137b65d424f5fb14cfdbaa1819e8d" candidate_uuid: "3216c8ce2e1e4a06b9f2eb16e42838f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beabd747c43e4c8d8c6d23b1a588befc" is_pre_election: true
I20260928 14:58:37.833949 11025 raft_consensus.cc:2454] T c06137b65d424f5fb14cfdbaa1819e8d P beabd747c43e4c8d8c6d23b1a588befc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3216c8ce2e1e4a06b9f2eb16e42838f1 in term 0.
I20260928 14:58:37.834322 11101 leader_election.cc:302] T c06137b65d424f5fb14cfdbaa1819e8d P 3216c8ce2e1e4a06b9f2eb16e42838f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f61915fb9ad4709874b6b96a0023381, 3216c8ce2e1e4a06b9f2eb16e42838f1; no voters:
I20260928 14:58:37.834749 11374 raft_consensus.cc:2781] T c06137b65d424f5fb14cfdbaa1819e8d P 3216c8ce2e1e4a06b9f2eb16e42838f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:58:37.834924 11374 raft_consensus.cc:482] T c06137b65d424f5fb14cfdbaa1819e8d P 3216c8ce2e1e4a06b9f2eb16e42838f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:58:37.835073 11374 raft_consensus.cc:3037] T c06137b65d424f5fb14cfdbaa1819e8d P 3216c8ce2e1e4a06b9f2eb16e42838f1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:37.837757 11374 raft_consensus.cc:504] T c06137b65d424f5fb14cfdbaa1819e8d P 3216c8ce2e1e4a06b9f2eb16e42838f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f61915fb9ad4709874b6b96a0023381" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 40889 } } peers { permanent_uuid: "beabd747c43e4c8d8c6d23b1a588befc" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 38393 } } peers { permanent_uuid: "3216c8ce2e1e4a06b9f2eb16e42838f1" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 36761 } }
I20260928 14:58:37.838402 11374 leader_election.cc:288] T c06137b65d424f5fb14cfdbaa1819e8d P 3216c8ce2e1e4a06b9f2eb16e42838f1 [CANDIDATE]: Term 1 election: Requested vote from peers 0f61915fb9ad4709874b6b96a0023381 (127.31.40.3:40889), beabd747c43e4c8d8c6d23b1a588befc (127.31.40.1:38393)
I20260928 14:58:37.838892 11302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c06137b65d424f5fb14cfdbaa1819e8d" candidate_uuid: "3216c8ce2e1e4a06b9f2eb16e42838f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f61915fb9ad4709874b6b96a0023381"
I20260928 14:58:37.839103 11302 raft_consensus.cc:3037] T c06137b65d424f5fb14cfdbaa1819e8d P 0f61915fb9ad4709874b6b96a0023381 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:37.839073 11025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c06137b65d424f5fb14cfdbaa1819e8d" candidate_uuid: "3216c8ce2e1e4a06b9f2eb16e42838f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beabd747c43e4c8d8c6d23b1a588befc"
I20260928 14:58:37.839303 11025 raft_consensus.cc:3037] T c06137b65d424f5fb14cfdbaa1819e8d P beabd747c43e4c8d8c6d23b1a588befc [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:37.841691 11302 raft_consensus.cc:2454] T c06137b65d424f5fb14cfdbaa1819e8d P 0f61915fb9ad4709874b6b96a0023381 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3216c8ce2e1e4a06b9f2eb16e42838f1 in term 1.
I20260928 14:58:37.842000 11025 raft_consensus.cc:2454] T c06137b65d424f5fb14cfdbaa1819e8d P beabd747c43e4c8d8c6d23b1a588befc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3216c8ce2e1e4a06b9f2eb16e42838f1 in term 1.
I20260928 14:58:37.842192 11101 leader_election.cc:302] T c06137b65d424f5fb14cfdbaa1819e8d P 3216c8ce2e1e4a06b9f2eb16e42838f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f61915fb9ad4709874b6b96a0023381, 3216c8ce2e1e4a06b9f2eb16e42838f1; no voters:
I20260928 14:58:37.842573 11374 raft_consensus.cc:2781] T c06137b65d424f5fb14cfdbaa1819e8d P 3216c8ce2e1e4a06b9f2eb16e42838f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:58:37.843106 11374 raft_consensus.cc:686] T c06137b65d424f5fb14cfdbaa1819e8d P 3216c8ce2e1e4a06b9f2eb16e42838f1 [term 1 LEADER]: Becoming Leader. State: Replica: 3216c8ce2e1e4a06b9f2eb16e42838f1, State: Running, Role: LEADER
I20260928 14:58:37.843629 11374 consensus_queue.cc:237] T c06137b65d424f5fb14cfdbaa1819e8d P 3216c8ce2e1e4a06b9f2eb16e42838f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f61915fb9ad4709874b6b96a0023381" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 40889 } } peers { permanent_uuid: "beabd747c43e4c8d8c6d23b1a588befc" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 38393 } } peers { permanent_uuid: "3216c8ce2e1e4a06b9f2eb16e42838f1" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 36761 } }
I20260928 14:58:37.849223 10857 catalog_manager.cc:5495] T c06137b65d424f5fb14cfdbaa1819e8d P 3216c8ce2e1e4a06b9f2eb16e42838f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3216c8ce2e1e4a06b9f2eb16e42838f1 (127.31.40.2). New cstate: current_term: 1 leader_uuid: "3216c8ce2e1e4a06b9f2eb16e42838f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f61915fb9ad4709874b6b96a0023381" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 40889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "beabd747c43e4c8d8c6d23b1a588befc" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 38393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3216c8ce2e1e4a06b9f2eb16e42838f1" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 36761 } health_report { overall_health: HEALTHY } } }
W20260928 14:58:37.898437 11214 tablet.cc:2369] T c06137b65d424f5fb14cfdbaa1819e8d P 3216c8ce2e1e4a06b9f2eb16e42838f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:58:37.925048 11363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:37.918328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40718 (local address 127.31.40.2:36761)
0928 14:58:37.918638 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:37.918643 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:37.918685 (+ 42us) server_negotiation.cc:407] Connection header received
0928 14:58:37.918926 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:37.918930 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:37.919033 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:37.919173 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:58:37.919541 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:37.920374 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:37.922347 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:37.922786 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:37.922906 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:37.923613 (+ 707us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:58:37.924315 (+ 702us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:58:37.924439 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:37.924583 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 14:58:37.924661 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148}
I20260928 14:58:37.934082 11025 raft_consensus.cc:1261] T c06137b65d424f5fb14cfdbaa1819e8d P beabd747c43e4c8d8c6d23b1a588befc [term 1 FOLLOWER]: Refusing update from remote peer 3216c8ce2e1e4a06b9f2eb16e42838f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:58:37.934098 11302 raft_consensus.cc:1261] T c06137b65d424f5fb14cfdbaa1819e8d P 0f61915fb9ad4709874b6b96a0023381 [term 1 FOLLOWER]: Refusing update from remote peer 3216c8ce2e1e4a06b9f2eb16e42838f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:58:37.935227 11377 consensus_queue.cc:1035] T c06137b65d424f5fb14cfdbaa1819e8d P 3216c8ce2e1e4a06b9f2eb16e42838f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f61915fb9ad4709874b6b96a0023381" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 40889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:58:37.935755 11374 consensus_queue.cc:1035] T c06137b65d424f5fb14cfdbaa1819e8d P 3216c8ce2e1e4a06b9f2eb16e42838f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beabd747c43e4c8d8c6d23b1a588befc" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 38393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:58:37.948594 11380 mvcc.cc:205] Tried to move back new op lower bound from 7334328393446514688 to 7334328393092415488. Current Snapshot: MvccSnapshot[applied={T|T < 7334328393446514688}]
I20260928 14:58:37.962522 10856 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:58:37.962977 10856 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:58:37.966967 10856 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a3b972e6f1bd40e38fc47fa4ec552e2a: Sending DeleteTablet for 3 replicas of tablet c06137b65d424f5fb14cfdbaa1819e8d
I20260928 14:58:37.968310 11282 tablet_service.cc:1514] Processing DeleteTablet for tablet c06137b65d424f5fb14cfdbaa1819e8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:58:37 UTC) from {username='kudu', principal='kudu/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:48880
I20260928 14:58:37.968278 11148 tablet_service.cc:1514] Processing DeleteTablet for tablet c06137b65d424f5fb14cfdbaa1819e8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:58:37 UTC) from {username='kudu', principal='kudu/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:40708
I20260928 14:58:37.969938 11005 tablet_service.cc:1514] Processing DeleteTablet for tablet c06137b65d424f5fb14cfdbaa1819e8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:58:37 UTC) from {username='kudu', principal='kudu/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:51726
I20260928 14:58:37.970068 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 10937
I20260928 14:58:37.970963 11392 tablet_replica.cc:330] T c06137b65d424f5fb14cfdbaa1819e8d P beabd747c43e4c8d8c6d23b1a588befc: stopping tablet replica
I20260928 14:58:37.971457 11392 raft_consensus.cc:2229] T c06137b65d424f5fb14cfdbaa1819e8d P beabd747c43e4c8d8c6d23b1a588befc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:58:37.972050 11392 raft_consensus.cc:2258] T c06137b65d424f5fb14cfdbaa1819e8d P beabd747c43e4c8d8c6d23b1a588befc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:58:37.974593 11392 ts_tablet_manager.cc:1905] T c06137b65d424f5fb14cfdbaa1819e8d P beabd747c43e4c8d8c6d23b1a588befc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:58:37.983279 11392 ts_tablet_manager.cc:1918] T c06137b65d424f5fb14cfdbaa1819e8d P beabd747c43e4c8d8c6d23b1a588befc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:58:37.983496 11392 log.cc:1196] T c06137b65d424f5fb14cfdbaa1819e8d P beabd747c43e4c8d8c6d23b1a588befc: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ts-0/wal/wals/c06137b65d424f5fb14cfdbaa1819e8d
I20260928 14:58:37.984153 11392 ts_tablet_manager.cc:1939] T c06137b65d424f5fb14cfdbaa1819e8d P beabd747c43e4c8d8c6d23b1a588befc: Deleting consensus metadata
I20260928 14:58:37.986161 10841 catalog_manager.cc:4841] TS beabd747c43e4c8d8c6d23b1a588befc (127.31.40.1:38393): tablet c06137b65d424f5fb14cfdbaa1819e8d (table test-table [id=8ece6eba5b4f4bc19bd01c01fc4ad697]) successfully deleted
I20260928 14:58:37.987660 11390 tablet_replica.cc:330] T c06137b65d424f5fb14cfdbaa1819e8d P 3216c8ce2e1e4a06b9f2eb16e42838f1: stopping tablet replica
I20260928 14:58:37.988194 11390 raft_consensus.cc:2229] T c06137b65d424f5fb14cfdbaa1819e8d P 3216c8ce2e1e4a06b9f2eb16e42838f1 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:58:37.988785 11390 raft_consensus.cc:2258] T c06137b65d424f5fb14cfdbaa1819e8d P 3216c8ce2e1e4a06b9f2eb16e42838f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:58:37.989559 11391 tablet_replica.cc:330] T c06137b65d424f5fb14cfdbaa1819e8d P 0f61915fb9ad4709874b6b96a0023381: stopping tablet replica
I20260928 14:58:37.989921 11391 raft_consensus.cc:2229] T c06137b65d424f5fb14cfdbaa1819e8d P 0f61915fb9ad4709874b6b96a0023381 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:58:37.990350 11391 raft_consensus.cc:2258] T c06137b65d424f5fb14cfdbaa1819e8d P 0f61915fb9ad4709874b6b96a0023381 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:58:37.992493 11391 ts_tablet_manager.cc:1905] T c06137b65d424f5fb14cfdbaa1819e8d P 0f61915fb9ad4709874b6b96a0023381: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:58:37.993491 11390 ts_tablet_manager.cc:1905] T c06137b65d424f5fb14cfdbaa1819e8d P 3216c8ce2e1e4a06b9f2eb16e42838f1: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 14:58:37 dist-test-slave-j968 krb5kdc[10380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.1@KRBTEST.COM
I20260928 14:58:37.999622 11391 ts_tablet_manager.cc:1918] T c06137b65d424f5fb14cfdbaa1819e8d P 0f61915fb9ad4709874b6b96a0023381: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:58:37.999792 11391 log.cc:1196] T c06137b65d424f5fb14cfdbaa1819e8d P 0f61915fb9ad4709874b6b96a0023381: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ts-2/wal/wals/c06137b65d424f5fb14cfdbaa1819e8d
I20260928 14:58:38.000304 11391 ts_tablet_manager.cc:1939] T c06137b65d424f5fb14cfdbaa1819e8d P 0f61915fb9ad4709874b6b96a0023381: Deleting consensus metadata
I20260928 14:58:38.001513 11390 ts_tablet_manager.cc:1918] T c06137b65d424f5fb14cfdbaa1819e8d P 3216c8ce2e1e4a06b9f2eb16e42838f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:58:38.001713 11390 log.cc:1196] T c06137b65d424f5fb14cfdbaa1819e8d P 3216c8ce2e1e4a06b9f2eb16e42838f1: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607365362981-31904-0/minicluster-data/ts-1/wal/wals/c06137b65d424f5fb14cfdbaa1819e8d
I20260928 14:58:38.002413 10841 catalog_manager.cc:4841] TS 0f61915fb9ad4709874b6b96a0023381 (127.31.40.3:40889): tablet c06137b65d424f5fb14cfdbaa1819e8d (table test-table [id=8ece6eba5b4f4bc19bd01c01fc4ad697]) successfully deleted
I20260928 14:58:38.002833 11390 ts_tablet_manager.cc:1939] T c06137b65d424f5fb14cfdbaa1819e8d P 3216c8ce2e1e4a06b9f2eb16e42838f1: Deleting consensus metadata
I20260928 14:58:38.004931 10842 catalog_manager.cc:4841] TS 3216c8ce2e1e4a06b9f2eb16e42838f1 (127.31.40.2:36761): tablet c06137b65d424f5fb14cfdbaa1819e8d (table test-table [id=8ece6eba5b4f4bc19bd01c01fc4ad697]) successfully deleted
I20260928 14:58:38.005726 11365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:37.971175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51728 (local address 127.31.40.1:38393)
0928 14:58:37.971372 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:37.971382 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:37.980976 (+ 9594us) server_negotiation.cc:407] Connection header received
0928 14:58:37.981462 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:37.981469 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:37.981569 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:37.981648 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:37.983637 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:37.984860 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:37.987146 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:37.987779 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:37.987927 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:37.996613 (+ 8686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:37.996632 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:37.996651 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:37.996704 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:38.000828 (+ 4124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:38.002167 (+ 1339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:38.002176 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:38.002179 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:38.002370 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:38.002876 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:38.002881 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:38.002885 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:38.003667 (+ 782us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:38.004268 (+ 601us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:38.004607 (+ 339us) server_negotiation.cc:299] Negotiation successful
0928 14:58:38.004982 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 14:58:38.006708 11065 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:58:38.132328 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 10937
I20260928 14:58:38.156577 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 11076
Sep 28 14:58:38 dist-test-slave-j968 krb5kdc[10380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.2@KRBTEST.COM
I20260928 14:58:38.176728 11363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:38.157600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40728 (local address 127.31.40.2:36761)
0928 14:58:38.157780 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:38.157784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:38.157803 (+ 19us) server_negotiation.cc:407] Connection header received
0928 14:58:38.157921 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:38.157924 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:38.157989 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:38.158073 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:38.160536 (+ 2463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:38.161373 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:38.163522 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:38.164041 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:38.164139 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:38.169888 (+ 5749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:38.169906 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:38.169924 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:38.169970 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:38.173836 (+ 3866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:38.174526 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:38.174535 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:38.174539 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:38.174683 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:38.175072 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:38.175077 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:38.175080 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:38.175447 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:38.175970 (+ 523us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:38.176143 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 14:58:38.176371 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 14:58:38.177493 11208 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:58:38.293901 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 11076
I20260928 14:58:38.318893 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 11219
Sep 28 14:58:38 dist-test-slave-j968 krb5kdc[10380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.3@KRBTEST.COM
I20260928 14:58:38.338320 11361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:38.319761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35500 (local address 127.31.40.3:40889)
0928 14:58:38.319986 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:38.319990 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:38.320087 (+ 97us) server_negotiation.cc:407] Connection header received
0928 14:58:38.320267 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:38.320271 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:38.320360 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:38.320446 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:38.322351 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:38.323240 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:38.325384 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:38.325794 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:38.325903 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:38.331349 (+ 5446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:38.331366 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:38.331384 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:38.331447 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:38.334942 (+ 3495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:38.335768 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:38.335778 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:38.335780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:38.336042 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:38.336590 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:38.336595 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:38.336597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:38.337142 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:38.337389 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:38.337717 (+ 328us) server_negotiation.cc:299] Negotiation successful
0928 14:58:38.337948 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 14:58:38.339054 11342 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:58:38.437577 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 11219
I20260928 14:58:38.455114 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 10825
I20260928 14:58:38.456562 10887 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:58:38.577363 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 10825
I20260928 14:58:38.601740 31904 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 14:58:38.937777 31904 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 14:58:38.937989 31904 mini_ranger.cc:66] Stopping Ranger...
I20260928 14:58:39.284122 31904 mini_ranger.cc:68] Stopped Ranger
2026-09-28 14:58:39.284 UTC [10410] LOG: received smart shutdown request
2026-09-28 14:58:39.288 UTC [10410] LOG: background worker "logical replication launcher" (PID 10419) exited with exit code 1
2026-09-28 14:58:39.288 UTC [10414] LOG: shutting down
2026-09-28 14:58:39.491 UTC [10410] LOG: database system is shut down
2026-09-28T14:58:39Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (52017 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:58:39 dist-test-slave-j968 krb5kdc[11406](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:58:39 dist-test-slave-j968 krb5kdc[11406](info): set up 2 sockets
Sep 28 14:58:39 dist-test-slave-j968 krb5kdc[11406](info): commencing operation
krb5kdc: starting...
W20260928 14:58:40.745287 31904 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Sep 28 14:58:40 dist-test-slave-j968 krb5kdc[11406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:58:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:58:40Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.31.40.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.31.40.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.31.40.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/krb5kdc/rangeradmin_127.31.40.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.31.40.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/krb5kdc/rangeradmin_127.31.40.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.31.40.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.31.40.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.31.40.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/krb5kdc/rangerlookup_127.31.40.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.31.40.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/krb5kdc/rangerlookup_127.31.40.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.40.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.31.40.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.40.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/krb5kdc/HTTP_127.31.40.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.40.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/krb5kdc/HTTP_127.31.40.20@KRBTEST.COM.keytab.
I20260928 14:58:41.078833 31904 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskBx3nvY/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/postgres -l logfile start
2026-09-28 14:58:43.680 UTC [11436] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 14:58:43.680 UTC [11436] LOG: listening on IPv4 address "127.31.40.20", port 41677
2026-09-28 14:58:43.683 UTC [11436] LOG: listening on Unix socket "/tmp/.s.PGSQL.41677"
2026-09-28 14:58:43.707 UTC [11439] LOG: database system was shut down at 2026-09-28 14:58:42 UTC
2026-09-28 14:58:43.712 UTC [11436] LOG: database system is ready to accept connections
I20260928 14:58:44.697151 31904 mini_postgres.cc:96] Postgres bound to 41677
2026-09-28 14:58:44.715 UTC [11447] FATAL: database "slave" does not exist
127.31.40.20:41677 - accepting connections
I20260928 14:58:44.715938 31904 mini_ranger.cc:161] Starting Ranger...
I20260928 14:58:44.743305 31904 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 14:58:45.297776 31904 mini_ranger.cc:87] Created ranger Postgres database
I20260928 14:58:45.297905 31904 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ranger-admin
2026-09-28 14:58:45,610 [I] DB FLAVOR :POSTGRES
2026-09-28 14:58:45,610 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 14:58:45,611 [I] Checking connection..
2026-09-28 14:58:45,611 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBx3nvY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBx3nvY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.40.20:41677/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 14:58:45,874 [I] Checking connection passed.
2026-09-28 14:58:45,875 [I] --------- Verifying version history table ---------
2026-09-28 14:58:45,875 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBx3nvY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBx3nvY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.40.20:41677/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 14:58:46,138 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 14:58:46,139 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBx3nvY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBx3nvY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.40.20:41677/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 14:58:46,387 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 14:58:46,387 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 14:58:46,387 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBx3nvY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBx3nvY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.40.20:41677/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBx3nvY/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 14:58:46.651 UTC [11511] WARNING: there is no transaction in progress
2026-09-28 14:58:46,657 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 14:58:46,658 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBx3nvY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBx3nvY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.40.20:41677/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 14:58:46,894 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 14:58:46,894 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 14:58:46,895 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBx3nvY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBx3nvY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.40.20:41677/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 14:58:47,152 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBx3nvY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBx3nvY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.40.20:41677/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 14:58:47,386 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBx3nvY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBx3nvY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.40.20:41677/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-j968.c.gcp-upstream.internal','N') ;"
2026-09-28 14:58:47,632 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBx3nvY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBx3nvY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.40.20:41677/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 14:58:47,899 [I] Table x_portal_user does not exist in database ranger
2026-09-28 14:58:47,899 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBx3nvY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBx3nvY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.40.20:41677/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 14:58:48,155 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 14:58:48,155 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBx3nvY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBx3nvY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.40.20:41677/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 14:58:48,384 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBx3nvY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBx3nvY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.40.20:41677/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 14:58:48,644 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 14:58:48,644 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBx3nvY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBx3nvY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.40.20:41677/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBx3nvY/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 14:58:49.648 UTC [11619] WARNING: there is no transaction in progress
2026-09-28 14:58:49.665 UTC [11619] WARNING: there is no transaction in progress
2026-09-28 14:58:49.683 UTC [11619] WARNING: there is no transaction in progress
2026-09-28 14:58:49.700 UTC [11619] WARNING: there is no transaction in progress
2026-09-28 14:58:49.717 UTC [11619] WARNING: there is no transaction in progress
2026-09-28 14:58:49.736 UTC [11619] WARNING: there is no transaction in progress
2026-09-28 14:58:49.804 UTC [11619] WARNING: there is no transaction in progress
2026-09-28 14:58:49.812 UTC [11619] WARNING: there is no transaction in progress
2026-09-28 14:58:49.827 UTC [11619] WARNING: there is no transaction in progress
2026-09-28 14:58:49.838 UTC [11619] WARNING: there is no transaction in progress
2026-09-28 14:58:49.850 UTC [11619] WARNING: there is no transaction in progress
2026-09-28 14:58:49.864 UTC [11619] WARNING: there is no transaction in progress
2026-09-28 14:58:49.875 UTC [11619] WARNING: there is no transaction in progress
2026-09-28 14:58:49.882 UTC [11619] WARNING: there is no transaction in progress
2026-09-28 14:58:49.890 UTC [11619] WARNING: there is no transaction in progress
2026-09-28 14:58:49.897 UTC [11619] WARNING: there is no transaction in progress
2026-09-28 14:58:50,605 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 14:58:50,605 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBx3nvY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBx3nvY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.40.20:41677/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 14:58:50,853 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 14:58:50,854 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBx3nvY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBx3nvY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.40.20:41677/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 14:58:51,123 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 14:58:51,124 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBx3nvY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBx3nvY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.40.20:41677/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 14:58:51,356 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 14:58:51,356 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBx3nvY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBx3nvY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.40.20:41677/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 14:58:51,620 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBx3nvY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBx3nvY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.40.20:41677/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 14:58:51,887 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBx3nvY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBx3nvY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.40.20:41677/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-j968.c.gcp-upstream.internal';"
2026-09-28 14:58:52,122 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 14:58:52,122 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBx3nvY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBx3nvY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.40.20:41677/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 14:58:52,362 [I] DB_PATCHES have already been applied
I20260928 14:58:52.368295 31904 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ranger-admin:/tmp/dist-test-taskBx3nvY/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskBx3nvY/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskBx3nvY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBx3nvY/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 14:58:52.368382 31904 mini_ranger.cc:193] Using host: 127.31.40.20
I20260928 14:58:52.383329 31904 mini_ranger.cc:227] Ranger admin URL: http://127.31.40.20:34877
Sep 28, 2026 2:58:52 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:58:52 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:58:52 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 2:58:52 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 2:58:52 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 2:58:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 2:58:53 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 14:58:53.567680 31904 mini_ranger.cc:160] Time spent starting Ranger: real 8.852s user 0.000s sys 0.040s
Sep 28, 2026 2:59:13 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 2:59:13 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 2:59:13 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 2:59:13 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 2:59:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 2:59:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 2:59:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 2:59:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 2:59:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 2:59:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 2:59:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 2:59:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 2:59:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 2:59:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 2:59:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 2:59:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 2:59:13 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 2:59:14 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 14:59:14.892885 31904 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.31.40.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.31.40.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.31.40.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/krb5kdc/rangerkms_127.31.40.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.31.40.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/krb5kdc/rangerkms_127.31.40.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.40.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.31.40.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.40.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/krb5kdc/HTTP_127.31.40.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.40.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/krb5kdc/HTTP_127.31.40.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 14:59:15 dist-test-slave-j968 krb5kdc[11406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607555, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 14:59:15.076308 31904 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 14:59:15.099647 31904 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 14:59:16.065311 31904 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 14:59:16.065428 31904 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ranger-kms
I20260928 14:59:16.065482 31904 mini_ranger_kms.cc:227] Using postgres at 127.31.40.20:41677
2026-09-28 14:59:16,416 [I] DB FLAVOR :POSTGRES
2026-09-28 14:59:16,416 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 14:59:16,416 [I] Checking connection
2026-09-28 14:59:16,416 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBx3nvY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.40.20:41677/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 14:59:16,679 [I] connection success
2026-09-28 14:59:16,680 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 14:59:16,680 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 14:59:16,680 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBx3nvY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.40.20:41677/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 14:59:16,927 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 14:59:16,927 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 14:59:16,927 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 14:59:16,927 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBx3nvY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.40.20:41677/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 14:59:17,216 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 14:59:17.734246 31904 mini_ranger_kms.cc:310] Created kms service
I20260928 14:59:17.918165 31904 mini_ranger_kms.cc:326] Created kudu user
I20260928 14:59:17.970067 31904 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 14:59:17 dist-test-slave-j968 krb5kdc[11406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607555, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.31.40.20@KRBTEST.COM
I20260928 14:59:18.378108 31904 mini_ranger_kms.cc:384] Added ranger policy
I20260928 14:59:18.378438 31904 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ranger-kms:/tmp/dist-test-taskBx3nvY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBx3nvY/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskBx3nvY/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 14:59:18.378481 31904 mini_ranger_kms.cc:240] Using host: 127.31.40.20
I20260928 14:59:18.397186 31904 mini_ranger_kms.cc:276] Ranger KMS PID: 11799
I20260928 14:59:18.397324 31904 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.31.40.20:58001
Sep 28, 2026 2:59:19 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:59:19 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:59:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 2:59:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 2:59:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 2:59:20 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 28 14:59:20 dist-test-slave-j968 krb5kdc[11406](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790607560, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.40.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:59:20.809803 31904 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.734s user 0.080s sys 0.295s
I20260928 14:59:20.810063 31904 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 14:59:20.810246 31904 mini_ranger_kms.cc:401] 127.31.40.20:58001/kms/v1/keys
Sep 28 14:59:21 dist-test-slave-j968 krb5kdc[11406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607560, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.40.20@KRBTEST.COM for HTTP/127.31.40.20@KRBTEST.COM
Sep 28 14:59:21 dist-test-slave-j968 krb5kdc[11406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607560, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.40.20@KRBTEST.COM for HTTP/127.31.40.20@KRBTEST.COM
Sep 28 14:59:23 dist-test-slave-j968 krb5kdc[11406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.31.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab.
Entry for principal kudu/127.31.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab.
I20260928 14:59:24.023252 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:46757
--webserver_interface=127.31.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:35347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.40.62:46757
--ranger_config_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.40.20:58001/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:59:24.369046 11850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:59:24.369343 11850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:59:24.369483 11850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:59:24.377040 11850 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:59:24.377157 11850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:59:24.377224 11850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:59:24.377326 11850 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:59:24.377408 11850 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:59:24.387916 11850 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:35347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/master-0/wal
--ranger_kms_url=127.31.40.20:58001/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.31.40.62:46757
--ranger_config_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:46757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:59:24.389331 11850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:59:24.390754 11850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:59:24.405299 11855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:59:24.405364 11858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:59:24.405344 11856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:59:24.406131 11850 server_base.cc:1034] running on GCE node
I20260928 14:59:24.407011 11850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:59:24.408679 11850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:59:24.409907 11850 hybrid_clock.cc:648] HybridClock initialized: now 1790607564409845 us; error 52 us; skew 500 ppm
I20260928 14:59:24.410288 11850 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:59:24 dist-test-slave-j968 krb5kdc[11406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607564, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:59:24.419624 11850 init.cc:377] Logged in from keytab as kudu/127.31.40.62@KRBTEST.COM (short username kudu)
I20260928 14:59:24.421515 11850 webserver.cc:460] Webserver started at http://127.31.40.62:37395/ using document root <none> and password file <none>
I20260928 14:59:24.422266 11850 fs_manager.cc:359] Metadata directory not provided
I20260928 14:59:24.422426 11850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:59:24.422761 11850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 14:59:24 dist-test-slave-j968 krb5kdc[11406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607564, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.62@KRBTEST.COM for HTTP/127.31.40.20@KRBTEST.COM
I20260928 14:59:24.483243 11850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/master-0/data/instance:
uuid: "e82e9783177b407fba3ef6b25c48b2cc"
format_stamp: "Formatted at 2026-09-28 14:59:24 on dist-test-slave-j968"
server_key: "d8f02cf2d90d4a71235a368e5d2cbeb9"
server_key_iv: "d5f96510218f5d4db2f9396d018e44e0"
server_key_version: "kuduclusterkey@0"
I20260928 14:59:24.484383 11850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/master-0/wal/instance:
uuid: "e82e9783177b407fba3ef6b25c48b2cc"
format_stamp: "Formatted at 2026-09-28 14:59:24 on dist-test-slave-j968"
server_key: "d8f02cf2d90d4a71235a368e5d2cbeb9"
server_key_iv: "d5f96510218f5d4db2f9396d018e44e0"
server_key_version: "kuduclusterkey@0"
I20260928 14:59:24.491427 11850 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260928 14:59:24.509178 11865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:59:24.510767 11850 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260928 14:59:24.510950 11850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/master-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/master-0/wal
uuid: "e82e9783177b407fba3ef6b25c48b2cc"
format_stamp: "Formatted at 2026-09-28 14:59:24 on dist-test-slave-j968"
server_key: "d8f02cf2d90d4a71235a368e5d2cbeb9"
server_key_iv: "d5f96510218f5d4db2f9396d018e44e0"
server_key_version: "kuduclusterkey@0"
I20260928 14:59:24.511188 11850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:59:24.535394 11850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:59:24.539806 11850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:59:24.540195 11850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:59:24.560295 11850 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.62:46757
I20260928 14:59:24.560313 11916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.62:46757 every 8 connection(s)
I20260928 14:59:24.561801 11850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
I20260928 14:59:24.566033 11917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:59:24.569121 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 11850
I20260928 14:59:24.569344 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/master-0/wal/instance
I20260928 14:59:24.570042 31904 external_mini_cluster.cc:1428] Setting key f2da06d8f327605b09701ca477069493
I20260928 14:59:24.579687 11917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e82e9783177b407fba3ef6b25c48b2cc: Bootstrap starting.
I20260928 14:59:24.583314 11917 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e82e9783177b407fba3ef6b25c48b2cc: Neither blocks nor log segments found. Creating new log.
I20260928 14:59:24.584390 11917 log.cc:824] T 00000000000000000000000000000000 P e82e9783177b407fba3ef6b25c48b2cc: Log is configured to *not* fsync() on all Append() calls
Sep 28 14:59:24 dist-test-slave-j968 krb5kdc[11406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:59:24.587584 11917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e82e9783177b407fba3ef6b25c48b2cc: No bootstrap required, opened a new log
I20260928 14:59:24.594247 11920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:24.572253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57938 (local address 127.31.40.62:46757)
0928 14:59:24.573048 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:24.573063 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:24.573114 (+ 51us) server_negotiation.cc:407] Connection header received
0928 14:59:24.575144 (+ 2030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:24.575184 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:24.575782 (+ 598us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:24.576289 (+ 507us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:59:24.577518 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:24.579563 (+ 2045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:24.581017 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:24.582163 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:24.582358 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:24.586718 (+ 4360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:59:24.586726 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:59:24.586741 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:59:24.586777 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:59:24.589919 (+ 3142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:24.590805 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:24.590813 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:24.590819 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:24.591014 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:24.591416 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:24.591419 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:24.591421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:24.592053 (+ 632us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:59:24.592301 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:24.592748 (+ 447us) server_negotiation.cc:299] Negotiation successful
0928 14:59:24.593193 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":333,"threads_started":1}
I20260928 14:59:24.596830 11917 raft_consensus.cc:348] T 00000000000000000000000000000000 P e82e9783177b407fba3ef6b25c48b2cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e82e9783177b407fba3ef6b25c48b2cc" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 46757 } }
I20260928 14:59:24.597375 11917 raft_consensus.cc:374] T 00000000000000000000000000000000 P e82e9783177b407fba3ef6b25c48b2cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:59:24.597510 11917 raft_consensus.cc:729] T 00000000000000000000000000000000 P e82e9783177b407fba3ef6b25c48b2cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e82e9783177b407fba3ef6b25c48b2cc, State: Initialized, Role: FOLLOWER
I20260928 14:59:24.598050 11917 consensus_queue.cc:260] T 00000000000000000000000000000000 P e82e9783177b407fba3ef6b25c48b2cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e82e9783177b407fba3ef6b25c48b2cc" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 46757 } }
I20260928 14:59:24.598245 11917 raft_consensus.cc:388] T 00000000000000000000000000000000 P e82e9783177b407fba3ef6b25c48b2cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:59:24.598340 11917 raft_consensus.cc:482] T 00000000000000000000000000000000 P e82e9783177b407fba3ef6b25c48b2cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:59:24.598464 11917 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e82e9783177b407fba3ef6b25c48b2cc [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:59:24.600587 11917 raft_consensus.cc:504] T 00000000000000000000000000000000 P e82e9783177b407fba3ef6b25c48b2cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e82e9783177b407fba3ef6b25c48b2cc" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 46757 } }
I20260928 14:59:24.601080 11917 leader_election.cc:302] T 00000000000000000000000000000000 P e82e9783177b407fba3ef6b25c48b2cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e82e9783177b407fba3ef6b25c48b2cc; no voters:
I20260928 14:59:24.601735 11917 leader_election.cc:288] T 00000000000000000000000000000000 P e82e9783177b407fba3ef6b25c48b2cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:59:24.601853 11922 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e82e9783177b407fba3ef6b25c48b2cc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:59:24.602552 11922 raft_consensus.cc:686] T 00000000000000000000000000000000 P e82e9783177b407fba3ef6b25c48b2cc [term 1 LEADER]: Becoming Leader. State: Replica: e82e9783177b407fba3ef6b25c48b2cc, State: Running, Role: LEADER
I20260928 14:59:24.603178 11922 consensus_queue.cc:237] T 00000000000000000000000000000000 P e82e9783177b407fba3ef6b25c48b2cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e82e9783177b407fba3ef6b25c48b2cc" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 46757 } }
I20260928 14:59:24.604122 11917 sys_catalog.cc:564] T 00000000000000000000000000000000 P e82e9783177b407fba3ef6b25c48b2cc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:59:24.606082 11923 sys_catalog.cc:455] T 00000000000000000000000000000000 P e82e9783177b407fba3ef6b25c48b2cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e82e9783177b407fba3ef6b25c48b2cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e82e9783177b407fba3ef6b25c48b2cc" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 46757 } } }
I20260928 14:59:24.606664 11923 sys_catalog.cc:458] T 00000000000000000000000000000000 P e82e9783177b407fba3ef6b25c48b2cc [sys.catalog]: This master's current role is: LEADER
I20260928 14:59:24.606492 11924 sys_catalog.cc:455] T 00000000000000000000000000000000 P e82e9783177b407fba3ef6b25c48b2cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader e82e9783177b407fba3ef6b25c48b2cc. Latest consensus state: current_term: 1 leader_uuid: "e82e9783177b407fba3ef6b25c48b2cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e82e9783177b407fba3ef6b25c48b2cc" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 46757 } } }
I20260928 14:59:24.606817 11924 sys_catalog.cc:458] T 00000000000000000000000000000000 P e82e9783177b407fba3ef6b25c48b2cc [sys.catalog]: This master's current role is: LEADER
I20260928 14:59:24.608072 11929 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:59:24.613411 11929 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:59:24.614125 11917 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 14:59:24.631026 11929 catalog_manager.cc:1347] Generated new cluster ID: 258f83965fda41a2ba68c9e974cd6ab2
I20260928 14:59:24.631220 11929 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:59:24.644270 11929 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:59:24.646251 11929 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:59:24.657168 11929 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e82e9783177b407fba3ef6b25c48b2cc: Generated new TSK 0
I20260928 14:59:24.658135 11929 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.40.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab.
Entry for principal kudu/127.31.40.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab.
I20260928 14:59:25.456856 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.40.20:58001/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.1:0
--local_ip_for_outbound_sockets=127.31.40.1
--webserver_interface=127.31.40.1
--webserver_port=0
--tserver_master_addrs=127.31.40.62:46757
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab
--principal=kudu/127.31.40.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:35347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 14:59:25 dist-test-slave-j968 krb5kdc[11406](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790607565, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 14:59:25.872509 11962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:59:25.873044 11962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:59:25.873245 11962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:59:25.882196 11962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:59:25.882360 11962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:59:25.882548 11962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.1
I20260928 14:59:25.894624 11962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:35347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.31.40.20:58001/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab
--principal=kudu/127.31.40.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.40.1
--webserver_port=0
--tserver_master_addrs=127.31.40.62:46757
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.1
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:59:25.896350 11962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:59:25.897861 11962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:59:25.921470 11972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:59:25.922439 11973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:59:25.930260 11975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:59:25.931519 11962 server_base.cc:1034] running on GCE node
I20260928 14:59:25.932363 11962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:59:25.933568 11962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:59:25.934952 11962 hybrid_clock.cc:648] HybridClock initialized: now 1790607565934805 us; error 66 us; skew 500 ppm
I20260928 14:59:25.935415 11962 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:59:25 dist-test-slave-j968 krb5kdc[11406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607565, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:59:25.939795 11962 init.cc:377] Logged in from keytab as kudu/127.31.40.1@KRBTEST.COM (short username kudu)
I20260928 14:59:25.952395 11962 webserver.cc:460] Webserver started at http://127.31.40.1:38331/ using document root <none> and password file <none>
I20260928 14:59:25.953299 11962 fs_manager.cc:359] Metadata directory not provided
I20260928 14:59:25.953469 11962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:59:25.953899 11962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 14:59:25 dist-test-slave-j968 krb5kdc[11406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607565, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.1@KRBTEST.COM for HTTP/127.31.40.20@KRBTEST.COM
I20260928 14:59:25.982084 11962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-0/data/instance:
uuid: "07672459d9cc4841aa75d13bbdfcfc6d"
format_stamp: "Formatted at 2026-09-28 14:59:25 on dist-test-slave-j968"
server_key: "67c8cfaf316094107f3ce6eca42dad67"
server_key_iv: "aef24fad6ea0adf7d3f39d3dc1f5be74"
server_key_version: "kuduclusterkey@0"
I20260928 14:59:25.983956 11962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-0/wal/instance:
uuid: "07672459d9cc4841aa75d13bbdfcfc6d"
format_stamp: "Formatted at 2026-09-28 14:59:25 on dist-test-slave-j968"
server_key: "67c8cfaf316094107f3ce6eca42dad67"
server_key_iv: "aef24fad6ea0adf7d3f39d3dc1f5be74"
server_key_version: "kuduclusterkey@0"
I20260928 14:59:26.000869 11962 fs_manager.cc:691] Time spent creating directory manager: real 0.016s user 0.005s sys 0.007s
I20260928 14:59:26.044651 11982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:59:26.047205 11962 fs_manager.cc:725] Time spent opening block manager: real 0.019s user 0.007s sys 0.011s
I20260928 14:59:26.047540 11962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-0/wal
uuid: "07672459d9cc4841aa75d13bbdfcfc6d"
format_stamp: "Formatted at 2026-09-28 14:59:25 on dist-test-slave-j968"
server_key: "67c8cfaf316094107f3ce6eca42dad67"
server_key_iv: "aef24fad6ea0adf7d3f39d3dc1f5be74"
server_key_version: "kuduclusterkey@0"
I20260928 14:59:26.047964 11962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:59:26.070011 11962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:59:26.075901 11962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:59:26.076339 11962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:59:26.080338 11962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:59:26.085003 11962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:59:26.085228 11962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260928 14:59:26.085496 11962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:59:26.085661 11962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 14:59:26 dist-test-slave-j968 krb5kdc[11406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607565, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.62@KRBTEST.COM for HTTP/127.31.40.20@KRBTEST.COM
I20260928 14:59:26.445995 11962 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.1:46251
I20260928 14:59:26.446035 12094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.1:46251 every 8 connection(s)
I20260928 14:59:26.447772 11962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
I20260928 14:59:26.451072 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 11962
I20260928 14:59:26.451270 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-0/wal/instance
I20260928 14:59:26.452250 31904 external_mini_cluster.cc:1428] Setting key 4de2e5851b4abe3a5516ccc68e07874d
Sep 28 14:59:26 dist-test-slave-j968 krb5kdc[11406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607565, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.1@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.40.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.2@KRBTEST.COM" created.
I20260928 14:59:26.487900 12098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:26.452160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:53133 (local address 127.31.40.62:46757)
0928 14:59:26.465405 (+ 13245us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:26.465411 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:26.465435 (+ 24us) server_negotiation.cc:407] Connection header received
0928 14:59:26.468133 (+ 2698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:26.468139 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:26.468244 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:26.468323 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:59:26.470955 (+ 2632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:26.472165 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:26.474700 (+ 2535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:26.475355 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:26.475465 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:26.480386 (+ 4921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:59:26.480394 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:59:26.480399 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:59:26.480444 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:59:26.484086 (+ 3642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:26.485004 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:26.485010 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:26.485014 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:26.485176 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:26.485691 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:26.485696 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:26.485699 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:26.486110 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:59:26.486255 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:26.487239 (+ 984us) server_negotiation.cc:299] Negotiation successful
0928 14:59:26.487487 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13103,"thread_start_us":13124,"threads_started":1}
I20260928 14:59:26.490473 12097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:26.451453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:46757 (local address 127.31.40.1:53133)
0928 14:59:26.464710 (+ 13257us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:26.464776 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0928 14:59:26.467730 (+ 2954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:59:26.469156 (+ 1426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:59:26.469170 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:59:26.469832 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:59:26.470721 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:26.470751 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:26.472451 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:26.472457 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:26.474508 (+ 2051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:26.474533 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:26.475898 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:26.475906 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:26.476208 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:26.477130 (+ 922us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:59:26.477160 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:59:26.480080 (+ 2920us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:59:26.484240 (+ 4160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:59:26.484250 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:59:26.484270 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:59:26.484857 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:59:26.485305 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:59:26.485309 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:59:26.485313 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:59:26.485565 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:59:26.486339 (+ 774us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:59:26.486348 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:59:26.486802 (+ 454us) client_negotiation.cc:769] Sending connection context
0928 14:59:26.488451 (+ 1649us) client_negotiation.cc:240] Negotiation successful
0928 14:59:26.489031 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12898,"thread_start_us":12660,"threads_started":1}
I20260928 14:59:26.492669 12095 heartbeater.cc:347] Connected to a master server at 127.31.40.62:46757
I20260928 14:59:26.492990 12095 heartbeater.cc:464] Registering TS with master...
I20260928 14:59:26.493888 12095 heartbeater.cc:511] Master 127.31.40.62:46757 requested a full tablet report, sending...
I20260928 14:59:26.496583 11882 ts_manager.cc:194] Registered new tserver with Master: 07672459d9cc4841aa75d13bbdfcfc6d (127.31.40.1:46251)
I20260928 14:59:26.498620 11882 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.1@KRBTEST.COM'} at 127.31.40.1:53133
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab.
Entry for principal kudu/127.31.40.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab.
I20260928 14:59:26.527633 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.40.20:58001/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.2:0
--local_ip_for_outbound_sockets=127.31.40.2
--webserver_interface=127.31.40.2
--webserver_port=0
--tserver_master_addrs=127.31.40.62:46757
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab
--principal=kudu/127.31.40.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:35347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 14:59:26 dist-test-slave-j968 krb5kdc[11406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607565, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.62@KRBTEST.COM for HTTP/127.31.40.20@KRBTEST.COM
W20260928 14:59:27.021072 12101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:59:27.021579 12101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:59:27.021802 12101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:59:27.031419 12101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:59:27.031587 12101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:59:27.031811 12101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.2
I20260928 14:59:27.046401 12101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:35347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.31.40.20:58001/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab
--principal=kudu/127.31.40.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.40.2
--webserver_port=0
--tserver_master_addrs=127.31.40.62:46757
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.2
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:59:27.047770 12101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:59:27.049126 12101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:59:27.065570 12115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:59:27.065611 12118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:59:27.065564 12116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:59:27.066824 12101 server_base.cc:1034] running on GCE node
I20260928 14:59:27.067484 12101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:59:27.068986 12101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:59:27.070353 12101 hybrid_clock.cc:648] HybridClock initialized: now 1790607567070215 us; error 130 us; skew 500 ppm
I20260928 14:59:27.070883 12101 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:59:27 dist-test-slave-j968 krb5kdc[11406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607567, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:59:27.075608 12101 init.cc:377] Logged in from keytab as kudu/127.31.40.2@KRBTEST.COM (short username kudu)
I20260928 14:59:27.081231 12101 webserver.cc:460] Webserver started at http://127.31.40.2:43659/ using document root <none> and password file <none>
I20260928 14:59:27.082031 12101 fs_manager.cc:359] Metadata directory not provided
I20260928 14:59:27.082176 12101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:59:27.082491 12101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 14:59:27 dist-test-slave-j968 krb5kdc[11406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607567, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.2@KRBTEST.COM for HTTP/127.31.40.20@KRBTEST.COM
I20260928 14:59:27.100757 12101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-1/data/instance:
uuid: "70933c12e76e4b5f900f69ccf6150213"
format_stamp: "Formatted at 2026-09-28 14:59:27 on dist-test-slave-j968"
server_key: "adb8eab0c24b9102a665bfe7e08ba16d"
server_key_iv: "f3a1b6763668375fce169bf762d0051a"
server_key_version: "kuduclusterkey@0"
I20260928 14:59:27.101665 12101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-1/wal/instance:
uuid: "70933c12e76e4b5f900f69ccf6150213"
format_stamp: "Formatted at 2026-09-28 14:59:27 on dist-test-slave-j968"
server_key: "adb8eab0c24b9102a665bfe7e08ba16d"
server_key_iv: "f3a1b6763668375fce169bf762d0051a"
server_key_version: "kuduclusterkey@0"
I20260928 14:59:27.107849 12101 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260928 14:59:27.132513 12125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:59:27.134426 12101 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.001s sys 0.006s
I20260928 14:59:27.134666 12101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-1/wal
uuid: "70933c12e76e4b5f900f69ccf6150213"
format_stamp: "Formatted at 2026-09-28 14:59:27 on dist-test-slave-j968"
server_key: "adb8eab0c24b9102a665bfe7e08ba16d"
server_key_iv: "f3a1b6763668375fce169bf762d0051a"
server_key_version: "kuduclusterkey@0"
I20260928 14:59:27.134938 12101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:59:27.168169 12101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:59:27.172974 12101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:59:27.173298 12101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:59:27.174414 12101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:59:27.176071 12101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:59:27.176170 12101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:59:27.176304 12101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:59:27.176378 12101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:59:27.221923 12101 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.2:40413
I20260928 14:59:27.221946 12237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.2:40413 every 8 connection(s)
I20260928 14:59:27.223384 12101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
I20260928 14:59:27.228832 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 12101
I20260928 14:59:27.229051 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-1/wal/instance
I20260928 14:59:27.229826 31904 external_mini_cluster.cc:1428] Setting key 8792c09ae861bb288c4f95cdcaa18b47
Sep 28 14:59:27 dist-test-slave-j968 krb5kdc[11406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607567, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.2@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.40.3@KRBTEST.COM; defaulting to no policy
I20260928 14:59:27.265232 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:27.230708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:38285 (local address 127.31.40.62:46757)
0928 14:59:27.231167 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:27.231175 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:27.243981 (+ 12806us) server_negotiation.cc:407] Connection header received
0928 14:59:27.244068 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:27.244072 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:27.244150 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:27.244224 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:59:27.245977 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:27.247229 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:27.250091 (+ 2862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:27.250852 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:27.251012 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:27.256122 (+ 5110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:59:27.256129 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:59:27.256133 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:59:27.256183 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:59:27.260227 (+ 4044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:27.261422 (+ 1195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:27.261431 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:27.261437 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:27.261608 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:27.262205 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:27.262211 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:27.262217 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:27.262624 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:59:27.262803 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:27.264526 (+ 1723us) server_negotiation.cc:299] Negotiation successful
0928 14:59:27.264828 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":842,"threads_started":1}
I20260928 14:59:27.266588 12240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:27.227656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:46757 (local address 127.31.40.2:38285)
0928 14:59:27.240583 (+ 12927us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:27.240652 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0928 14:59:27.243500 (+ 2848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:59:27.244657 (+ 1157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:59:27.244666 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:59:27.245120 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:59:27.245770 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:27.245789 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:27.247453 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:27.247461 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:27.249869 (+ 2408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:27.249899 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:27.251370 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:27.251380 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:27.251746 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:27.252641 (+ 895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:59:27.252677 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:59:27.255780 (+ 3103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:59:27.260447 (+ 4667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:59:27.260461 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:59:27.260484 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:59:27.261243 (+ 759us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:59:27.261752 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:59:27.261758 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:59:27.261782 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:59:27.262062 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:59:27.262885 (+ 823us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:59:27.262897 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:59:27.263425 (+ 528us) client_negotiation.cc:769] Sending connection context
0928 14:59:27.263775 (+ 350us) client_negotiation.cc:240] Negotiation successful
0928 14:59:27.265218 (+ 1443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12528,"thread_start_us":10976,"threads_started":1}
I20260928 14:59:27.268179 12238 heartbeater.cc:347] Connected to a master server at 127.31.40.62:46757
I20260928 14:59:27.268532 12238 heartbeater.cc:464] Registering TS with master...
I20260928 14:59:27.269366 12238 heartbeater.cc:511] Master 127.31.40.62:46757 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.3@KRBTEST.COM" created.
I20260928 14:59:27.271193 11882 ts_manager.cc:194] Registered new tserver with Master: 70933c12e76e4b5f900f69ccf6150213 (127.31.40.2:40413)
I20260928 14:59:27.272504 11882 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.2@KRBTEST.COM'} at 127.31.40.2:38285
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab.
Entry for principal kudu/127.31.40.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab.
I20260928 14:59:27.303256 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.40.20:58001/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.3:0
--local_ip_for_outbound_sockets=127.31.40.3
--webserver_interface=127.31.40.3
--webserver_port=0
--tserver_master_addrs=127.31.40.62:46757
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab
--principal=kudu/127.31.40.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:35347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:59:27.502517 12095 heartbeater.cc:503] Master 127.31.40.62:46757 was elected leader, sending a full tablet report...
W20260928 14:59:27.682457 12244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:59:27.682807 12244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:59:27.682984 12244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:59:27.690503 12244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:59:27.690675 12244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:59:27.690846 12244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.3
I20260928 14:59:27.701613 12244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:35347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.31.40.20:58001/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab
--principal=kudu/127.31.40.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.40.3
--webserver_port=0
--tserver_master_addrs=127.31.40.62:46757
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.3
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:59:27.703024 12244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:59:27.704618 12244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:59:27.719098 12250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:59:27.719247 12249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:59:27.719897 12252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:59:27.721807 12244 server_base.cc:1034] running on GCE node
I20260928 14:59:27.722605 12244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:59:27.723905 12244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:59:27.725152 12244 hybrid_clock.cc:648] HybridClock initialized: now 1790607567725073 us; error 67 us; skew 500 ppm
I20260928 14:59:27.725551 12244 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:59:27 dist-test-slave-j968 krb5kdc[11406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607567, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:59:27.729645 12244 init.cc:377] Logged in from keytab as kudu/127.31.40.3@KRBTEST.COM (short username kudu)
I20260928 14:59:27.731630 12244 webserver.cc:460] Webserver started at http://127.31.40.3:37761/ using document root <none> and password file <none>
I20260928 14:59:27.732559 12244 fs_manager.cc:359] Metadata directory not provided
I20260928 14:59:27.732693 12244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:59:27.733062 12244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 14:59:27 dist-test-slave-j968 krb5kdc[11406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607567, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.3@KRBTEST.COM for HTTP/127.31.40.20@KRBTEST.COM
I20260928 14:59:27.751158 12244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-2/data/instance:
uuid: "05bc129bf3744e87b1844b7447b39055"
format_stamp: "Formatted at 2026-09-28 14:59:27 on dist-test-slave-j968"
server_key: "98f6a999917a90b8b48af2771eaf9109"
server_key_iv: "99ec95b7f09068d10263016ad0ae6a61"
server_key_version: "kuduclusterkey@0"
I20260928 14:59:27.752164 12244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-2/wal/instance:
uuid: "05bc129bf3744e87b1844b7447b39055"
format_stamp: "Formatted at 2026-09-28 14:59:27 on dist-test-slave-j968"
server_key: "98f6a999917a90b8b48af2771eaf9109"
server_key_iv: "99ec95b7f09068d10263016ad0ae6a61"
server_key_version: "kuduclusterkey@0"
I20260928 14:59:27.758684 12244 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 14:59:27.780493 12259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:59:27.782351 12244 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.004s sys 0.000s
I20260928 14:59:27.782565 12244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-2/wal
uuid: "05bc129bf3744e87b1844b7447b39055"
format_stamp: "Formatted at 2026-09-28 14:59:27 on dist-test-slave-j968"
server_key: "98f6a999917a90b8b48af2771eaf9109"
server_key_iv: "99ec95b7f09068d10263016ad0ae6a61"
server_key_version: "kuduclusterkey@0"
I20260928 14:59:27.782819 12244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:59:27.826663 12244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:59:27.831250 12244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:59:27.831584 12244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:59:27.832744 12244 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:59:27.834460 12244 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:59:27.834566 12244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:59:27.834709 12244 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:59:27.834779 12244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:59:27.869768 12244 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.3:33363
I20260928 14:59:27.869808 12371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.3:33363 every 8 connection(s)
I20260928 14:59:27.871088 12244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
I20260928 14:59:27.877110 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 12244
I20260928 14:59:27.877313 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-2/wal/instance
I20260928 14:59:27.877882 31904 external_mini_cluster.cc:1428] Setting key b2dc83b3bb50ba929ea0d85d3485bb23
Sep 28 14:59:27 dist-test-slave-j968 krb5kdc[11406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607567, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.3@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:59:27.893456 12374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:27.873996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:52483 (local address 127.31.40.62:46757)
0928 14:59:27.874514 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:27.874518 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:27.875541 (+ 1023us) server_negotiation.cc:407] Connection header received
0928 14:59:27.877888 (+ 2347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:27.877893 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:27.878041 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:27.878136 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:59:27.880230 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:27.881301 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:27.883112 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:27.883597 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:27.883704 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:27.887389 (+ 3685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:59:27.887395 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:59:27.887398 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:59:27.887433 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:59:27.890126 (+ 2693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:27.890879 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:27.890883 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:27.890885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:27.891008 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:27.891457 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:27.891460 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:27.891462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:27.891754 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:59:27.891933 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:27.892869 (+ 936us) server_negotiation.cc:299] Negotiation successful
0928 14:59:27.893153 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":256,"threads_started":1}
I20260928 14:59:27.894599 12375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:27.874529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:46757 (local address 127.31.40.3:52483)
0928 14:59:27.875362 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:27.875403 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 14:59:27.877518 (+ 2115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:59:27.878439 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:59:27.878452 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:59:27.879066 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:59:27.880036 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:27.880063 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:27.881467 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:27.881471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:27.882962 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:27.882983 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:27.883909 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:27.883918 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:27.884134 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:27.884774 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:59:27.884856 (+ 82us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:59:27.887104 (+ 2248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:59:27.890267 (+ 3163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:59:27.890276 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:59:27.890296 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:59:27.890757 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:59:27.891128 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:59:27.891131 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:59:27.891134 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:59:27.891345 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:59:27.891951 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:59:27.891958 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:59:27.892429 (+ 471us) client_negotiation.cc:769] Sending connection context
0928 14:59:27.892739 (+ 310us) client_negotiation.cc:240] Negotiation successful
0928 14:59:27.893263 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":384,"threads_started":1}
I20260928 14:59:27.896080 12372 heartbeater.cc:347] Connected to a master server at 127.31.40.62:46757
I20260928 14:59:27.896392 12372 heartbeater.cc:464] Registering TS with master...
I20260928 14:59:27.897140 12372 heartbeater.cc:511] Master 127.31.40.62:46757 requested a full tablet report, sending...
I20260928 14:59:27.898561 11882 ts_manager.cc:194] Registered new tserver with Master: 05bc129bf3744e87b1844b7447b39055 (127.31.40.3:33363)
I20260928 14:59:27.899456 11882 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.3@KRBTEST.COM'} at 127.31.40.3:52483
I20260928 14:59:27.906406 31904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:59:27.908152 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 11962
Sep 28 14:59:27 dist-test-slave-j968 krb5kdc[11406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.1@KRBTEST.COM
I20260928 14:59:27.925763 12377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:27.909218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52746 (local address 127.31.40.1:46251)
0928 14:59:27.909949 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:27.909958 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:27.909987 (+ 29us) server_negotiation.cc:407] Connection header received
0928 14:59:27.910053 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:27.910059 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:27.910273 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:27.910454 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:59:27.912009 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:27.913035 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:27.914503 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:27.915113 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:27.915245 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:27.919696 (+ 4451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:59:27.919709 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:59:27.919725 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:59:27.919764 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:59:27.922737 (+ 2973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:27.923470 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:27.923477 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:27.923480 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:27.923628 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:27.924071 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:27.924076 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:27.924079 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:27.924472 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:59:27.924797 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:27.925118 (+ 321us) server_negotiation.cc:299] Negotiation successful
0928 14:59:27.925410 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":344,"threads_started":1}
I20260928 14:59:27.927169 12090 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:59:28.016266 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 11962
I20260928 14:59:28.033339 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 12101
Sep 28 14:59:28 dist-test-slave-j968 krb5kdc[11406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.2@KRBTEST.COM
I20260928 14:59:28.052145 12379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:28.034436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46694 (local address 127.31.40.2:40413)
0928 14:59:28.034975 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:28.034982 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:28.035008 (+ 26us) server_negotiation.cc:407] Connection header received
0928 14:59:28.035081 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:28.035087 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:28.035321 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:28.035462 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:59:28.037054 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:28.038059 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:28.039627 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:28.040480 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:28.040596 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:28.045317 (+ 4721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:59:28.045333 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:59:28.045351 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:59:28.045392 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:59:28.048792 (+ 3400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:28.049615 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:28.049623 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:28.049626 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:28.049768 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:28.050340 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:28.050344 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:28.050352 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:28.050750 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:59:28.051078 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:28.051421 (+ 343us) server_negotiation.cc:299] Negotiation successful
0928 14:59:28.051635 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":275,"threads_started":1}
I20260928 14:59:28.053503 12233 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:59:28.144506 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 12101
I20260928 14:59:28.166327 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 12244
Sep 28 14:59:28 dist-test-slave-j968 krb5kdc[11406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.3@KRBTEST.COM
I20260928 14:59:28.183816 12381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:28.167386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35226 (local address 127.31.40.3:33363)
0928 14:59:28.168145 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:28.168152 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:28.168182 (+ 30us) server_negotiation.cc:407] Connection header received
0928 14:59:28.168244 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:28.168250 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:28.168472 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:28.168599 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:59:28.169938 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:28.171033 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:28.172624 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:28.173199 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:28.173310 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:28.177738 (+ 4428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:59:28.177750 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:59:28.177766 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:59:28.177801 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:59:28.180933 (+ 3132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:28.181628 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:28.181636 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:28.181638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:28.181849 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:28.182189 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:28.182193 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:28.182195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:28.182559 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:59:28.182906 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:28.183294 (+ 388us) server_negotiation.cc:299] Negotiation successful
0928 14:59:28.183514 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":368,"threads_started":1}
I20260928 14:59:28.185263 12367 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:59:28.269464 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 12244
I20260928 14:59:28.286028 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 11850
I20260928 14:59:28.287384 11912 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:59:28.396703 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 11850
I20260928 14:59:28.419740 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:46757
--webserver_interface=127.31.40.62
--webserver_port=37395
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:35347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.40.62:46757
--ranger_config_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.40.20:58001/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.40.20:58001/kms with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:59:28.755260 12384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:59:28.755565 12384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:59:28.755728 12384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:59:28.763155 12384 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:59:28.763281 12384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:59:28.763371 12384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:59:28.763451 12384 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:59:28.763515 12384 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:59:28.774381 12384 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:35347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.40.20:58001/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.31.40.62:46757
--ranger_config_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:46757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.40.62
--webserver_port=37395
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:59:28.775832 12384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:59:28.777284 12384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:59:28.789638 12389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:59:28.789747 12392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:59:28.789705 12390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:59:28.790463 12384 server_base.cc:1034] running on GCE node
I20260928 14:59:28.791399 12384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:59:28.792954 12384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:59:28.794178 12384 hybrid_clock.cc:648] HybridClock initialized: now 1790607568794113 us; error 44 us; skew 500 ppm
I20260928 14:59:28.794589 12384 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:59:28 dist-test-slave-j968 krb5kdc[11406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607568, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:59:28.797856 12384 init.cc:377] Logged in from keytab as kudu/127.31.40.62@KRBTEST.COM (short username kudu)
I20260928 14:59:28.799371 12384 webserver.cc:460] Webserver started at http://127.31.40.62:37395/ using document root <none> and password file <none>
I20260928 14:59:28.800086 12384 fs_manager.cc:359] Metadata directory not provided
I20260928 14:59:28.800199 12384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 14:59:28 dist-test-slave-j968 krb5kdc[11406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607568, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.62@KRBTEST.COM for HTTP/127.31.40.20@KRBTEST.COM
I20260928 14:59:28.876072 12384 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 14:59:28.879739 12400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:59:28.881407 12384 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:59:28.881611 12384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/master-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/master-0/wal
uuid: "e82e9783177b407fba3ef6b25c48b2cc"
format_stamp: "Formatted at 2026-09-28 14:59:24 on dist-test-slave-j968"
server_key: "d8f02cf2d90d4a71235a368e5d2cbeb9"
server_key_iv: "d5f96510218f5d4db2f9396d018e44e0"
server_key_version: "kuduclusterkey@0"
I20260928 14:59:28.882638 12384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:59:28.906787 12384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:59:28.911320 12384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:59:28.911640 12384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:59:28.932612 12384 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.62:46757
I20260928 14:59:28.932590 12451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.62:46757 every 8 connection(s)
I20260928 14:59:28.934190 12384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
I20260928 14:59:28.934839 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 12384
I20260928 14:59:28.935953 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.40.20:58001/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.1:46251
--local_ip_for_outbound_sockets=127.31.40.1
--tserver_master_addrs=127.31.40.62:46757
--webserver_port=38331
--webserver_interface=127.31.40.1
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab
--principal=kudu/127.31.40.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:35347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.40.20:58001/kms with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:59:28.942495 12452 sys_catalog.cc:263] Verifying existing consensus state
I20260928 14:59:28.947978 12452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e82e9783177b407fba3ef6b25c48b2cc: Bootstrap starting.
I20260928 14:59:28.976467 12452 log.cc:824] T 00000000000000000000000000000000 P e82e9783177b407fba3ef6b25c48b2cc: Log is configured to *not* fsync() on all Append() calls
I20260928 14:59:28.988094 12452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e82e9783177b407fba3ef6b25c48b2cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:59:28.988689 12452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e82e9783177b407fba3ef6b25c48b2cc: Bootstrap complete.
I20260928 14:59:28.998438 12452 raft_consensus.cc:348] T 00000000000000000000000000000000 P e82e9783177b407fba3ef6b25c48b2cc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e82e9783177b407fba3ef6b25c48b2cc" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 46757 } }
I20260928 14:59:28.999330 12452 raft_consensus.cc:729] T 00000000000000000000000000000000 P e82e9783177b407fba3ef6b25c48b2cc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e82e9783177b407fba3ef6b25c48b2cc, State: Initialized, Role: FOLLOWER
I20260928 14:59:29.000075 12452 consensus_queue.cc:260] T 00000000000000000000000000000000 P e82e9783177b407fba3ef6b25c48b2cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e82e9783177b407fba3ef6b25c48b2cc" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 46757 } }
I20260928 14:59:29.000327 12452 raft_consensus.cc:388] T 00000000000000000000000000000000 P e82e9783177b407fba3ef6b25c48b2cc [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:59:29.000455 12452 raft_consensus.cc:482] T 00000000000000000000000000000000 P e82e9783177b407fba3ef6b25c48b2cc [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:59:29.000653 12452 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e82e9783177b407fba3ef6b25c48b2cc [term 1 FOLLOWER]: Advancing to term 2
I20260928 14:59:29.004639 12452 raft_consensus.cc:504] T 00000000000000000000000000000000 P e82e9783177b407fba3ef6b25c48b2cc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e82e9783177b407fba3ef6b25c48b2cc" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 46757 } }
I20260928 14:59:29.005169 12452 leader_election.cc:302] T 00000000000000000000000000000000 P e82e9783177b407fba3ef6b25c48b2cc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e82e9783177b407fba3ef6b25c48b2cc; no voters:
I20260928 14:59:29.005853 12452 leader_election.cc:288] T 00000000000000000000000000000000 P e82e9783177b407fba3ef6b25c48b2cc [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 14:59:29.006078 12456 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e82e9783177b407fba3ef6b25c48b2cc [term 2 FOLLOWER]: Leader election won for term 2
I20260928 14:59:29.007078 12456 raft_consensus.cc:686] T 00000000000000000000000000000000 P e82e9783177b407fba3ef6b25c48b2cc [term 2 LEADER]: Becoming Leader. State: Replica: e82e9783177b407fba3ef6b25c48b2cc, State: Running, Role: LEADER
I20260928 14:59:29.007766 12456 consensus_queue.cc:237] T 00000000000000000000000000000000 P e82e9783177b407fba3ef6b25c48b2cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e82e9783177b407fba3ef6b25c48b2cc" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 46757 } }
I20260928 14:59:29.007997 12452 sys_catalog.cc:564] T 00000000000000000000000000000000 P e82e9783177b407fba3ef6b25c48b2cc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:59:29.012207 12457 sys_catalog.cc:455] T 00000000000000000000000000000000 P e82e9783177b407fba3ef6b25c48b2cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e82e9783177b407fba3ef6b25c48b2cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e82e9783177b407fba3ef6b25c48b2cc" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 46757 } } }
I20260928 14:59:29.012671 12457 sys_catalog.cc:458] T 00000000000000000000000000000000 P e82e9783177b407fba3ef6b25c48b2cc [sys.catalog]: This master's current role is: LEADER
I20260928 14:59:29.013212 12458 sys_catalog.cc:455] T 00000000000000000000000000000000 P e82e9783177b407fba3ef6b25c48b2cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader e82e9783177b407fba3ef6b25c48b2cc. Latest consensus state: current_term: 2 leader_uuid: "e82e9783177b407fba3ef6b25c48b2cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e82e9783177b407fba3ef6b25c48b2cc" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 46757 } } }
I20260928 14:59:29.013523 12458 sys_catalog.cc:458] T 00000000000000000000000000000000 P e82e9783177b407fba3ef6b25c48b2cc [sys.catalog]: This master's current role is: LEADER
I20260928 14:59:29.021988 12464 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:59:29.029181 12464 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:59:29.030942 12464 catalog_manager.cc:1259] Loaded cluster ID: 258f83965fda41a2ba68c9e974cd6ab2
I20260928 14:59:29.031112 12464 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:59:29.031183 12452 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 14:59:29.036100 12464 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:59:29.038019 12464 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e82e9783177b407fba3ef6b25c48b2cc: Loaded TSK: 0
I20260928 14:59:29.046976 12464 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 14:59:29.488322 12454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:59:29.488894 12454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:59:29.489132 12454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:59:29.502687 12454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:59:29.502869 12454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:59:29.503158 12454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.1
I20260928 14:59:29.522032 12454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:35347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.40.20:58001/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab
--principal=kudu/127.31.40.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.1:46251
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.40.1
--webserver_port=38331
--tserver_master_addrs=127.31.40.62:46757
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.1
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:59:29.524323 12454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:59:29.526396 12454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:59:29.586571 12492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:59:29.586632 12491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:59:29.613991 12494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:59:29.614423 12454 server_base.cc:1034] running on GCE node
I20260928 14:59:29.615509 12454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:59:29.618460 12454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:59:29.619845 12454 hybrid_clock.cc:648] HybridClock initialized: now 1790607569619751 us; error 65 us; skew 500 ppm
I20260928 14:59:29.620323 12454 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:59:29 dist-test-slave-j968 krb5kdc[11406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607569, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:59:29.627487 12454 init.cc:377] Logged in from keytab as kudu/127.31.40.1@KRBTEST.COM (short username kudu)
I20260928 14:59:29.630698 12454 webserver.cc:460] Webserver started at http://127.31.40.1:38331/ using document root <none> and password file <none>
I20260928 14:59:29.631788 12454 fs_manager.cc:359] Metadata directory not provided
I20260928 14:59:29.632081 12454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 14:59:29 dist-test-slave-j968 krb5kdc[11406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607569, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.1@KRBTEST.COM for HTTP/127.31.40.20@KRBTEST.COM
I20260928 14:59:29.659497 12454 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 14:59:29.663182 12502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:59:29.665242 12454 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260928 14:59:29.665447 12454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-0/wal
uuid: "07672459d9cc4841aa75d13bbdfcfc6d"
format_stamp: "Formatted at 2026-09-28 14:59:25 on dist-test-slave-j968"
server_key: "67c8cfaf316094107f3ce6eca42dad67"
server_key_iv: "aef24fad6ea0adf7d3f39d3dc1f5be74"
server_key_version: "kuduclusterkey@0"
I20260928 14:59:29.666947 12454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:59:29.705700 12454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:59:29.710580 12454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:59:29.710919 12454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:59:29.712162 12454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:59:29.714082 12454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:59:29.714263 12454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:59:29.714471 12454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:59:29.714638 12454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:59:29.772442 12454 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.1:46251
I20260928 14:59:29.772555 12614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.1:46251 every 8 connection(s)
I20260928 14:59:29.774595 12454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
I20260928 14:59:29.784878 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 12454
I20260928 14:59:29.785753 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.40.20:58001/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.2:40413
--local_ip_for_outbound_sockets=127.31.40.2
--tserver_master_addrs=127.31.40.62:46757
--webserver_port=43659
--webserver_interface=127.31.40.2
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab
--principal=kudu/127.31.40.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:35347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.40.20:58001/kms with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 14:59:29 dist-test-slave-j968 krb5kdc[11406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607569, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.1@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:59:29.815335 12617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:29.778555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:46757 (local address 127.31.40.1:36357)
0928 14:59:29.779470 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:29.779521 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 14:59:29.781745 (+ 2224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:59:29.783352 (+ 1607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:59:29.783366 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:59:29.784025 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:59:29.784652 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:29.784675 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:29.787052 (+ 2377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:29.787062 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:29.788836 (+ 1774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:29.788852 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:29.790102 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:29.790111 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:29.790408 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:29.791175 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:59:29.791201 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:59:29.794492 (+ 3291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:59:29.808498 (+ 14006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:59:29.808514 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:59:29.808539 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:59:29.809457 (+ 918us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:59:29.810288 (+ 831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:59:29.810294 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:59:29.810298 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:59:29.810665 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:59:29.812070 (+ 1405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:59:29.812080 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:59:29.812611 (+ 531us) client_negotiation.cc:769] Sending connection context
0928 14:59:29.813017 (+ 406us) client_negotiation.cc:240] Negotiation successful
0928 14:59:29.813727 (+ 710us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":381,"threads_started":1}
I20260928 14:59:29.816792 12618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:29.778633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:36357 (local address 127.31.40.62:46757)
0928 14:59:29.780507 (+ 1874us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:29.780532 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:29.780589 (+ 57us) server_negotiation.cc:407] Connection header received
0928 14:59:29.782396 (+ 1807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:29.782410 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:29.782785 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:29.783196 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:59:29.784892 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:29.786871 (+ 1979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:29.788999 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:29.789820 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:29.790057 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:29.794862 (+ 4805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:59:29.794878 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:59:29.794903 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:59:29.794962 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:59:29.802389 (+ 7427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:29.809783 (+ 7394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:29.809797 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:29.809811 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:29.810094 (+ 283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:29.810860 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:29.810870 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:29.810877 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:29.811800 (+ 923us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:59:29.812116 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:29.813345 (+ 1229us) server_negotiation.cc:299] Negotiation successful
0928 14:59:29.813960 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1565,"thread_start_us":1316,"threads_started":1}
I20260928 14:59:29.820397 12615 heartbeater.cc:347] Connected to a master server at 127.31.40.62:46757
I20260928 14:59:29.820829 12615 heartbeater.cc:464] Registering TS with master...
I20260928 14:59:29.822013 12615 heartbeater.cc:511] Master 127.31.40.62:46757 requested a full tablet report, sending...
W20260928 14:59:29.826964 12615 heartbeater.cc:644] Failed to heartbeat to 127.31.40.62:46757 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 14:59:29.827379 12615 heartbeater.cc:464] Registering TS with master...
I20260928 14:59:29.827785 12615 heartbeater.cc:511] Master 127.31.40.62:46757 requested a full tablet report, sending...
I20260928 14:59:29.829344 12615 heartbeater.cc:464] Registering TS with master...
I20260928 14:59:29.829665 12615 heartbeater.cc:511] Master 127.31.40.62:46757 requested a full tablet report, sending...
W20260928 14:59:29.831067 12615 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 14:59:30.253957 12619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:59:30.254369 12619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:59:30.254570 12619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:59:30.266465 12619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:59:30.266700 12619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:59:30.266918 12619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.2
I20260928 14:59:30.282910 12619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:35347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.40.20:58001/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab
--principal=kudu/127.31.40.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.2:40413
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.40.2
--webserver_port=43659
--tserver_master_addrs=127.31.40.62:46757
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.2
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:59:30.284732 12619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:59:30.286473 12619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Sep 28 14:59:30 dist-test-slave-j968 krb5kdc[11406](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790607570, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 14:59:30.318454 12635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:59:30.318470 12636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:59:30.319742 12638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:59:30.320714 12619 server_base.cc:1034] running on GCE node
I20260928 14:59:30.321796 12619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:59:30.328447 12619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:59:30.331992 12619 hybrid_clock.cc:648] HybridClock initialized: now 1790607570331878 us; error 106 us; skew 500 ppm
I20260928 14:59:30.332414 12619 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:59:30 dist-test-slave-j968 krb5kdc[11406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607570, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:59:30.335978 12619 init.cc:377] Logged in from keytab as kudu/127.31.40.2@KRBTEST.COM (short username kudu)
I20260928 14:59:30.337961 12619 webserver.cc:460] Webserver started at http://127.31.40.2:43659/ using document root <none> and password file <none>
I20260928 14:59:30.338832 12619 fs_manager.cc:359] Metadata directory not provided
I20260928 14:59:30.338995 12619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 14:59:30 dist-test-slave-j968 krb5kdc[11406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607570, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.2@KRBTEST.COM for HTTP/127.31.40.20@KRBTEST.COM
I20260928 14:59:30.368520 12619 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.003s sys 0.004s
I20260928 14:59:30.374337 12646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:59:30.376540 12619 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.004s
I20260928 14:59:30.376761 12619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-1/wal
uuid: "70933c12e76e4b5f900f69ccf6150213"
format_stamp: "Formatted at 2026-09-28 14:59:27 on dist-test-slave-j968"
server_key: "adb8eab0c24b9102a665bfe7e08ba16d"
server_key_iv: "f3a1b6763668375fce169bf762d0051a"
server_key_version: "kuduclusterkey@0"
I20260928 14:59:30.378293 12619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:59:30.408084 12619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:59:30.413714 12619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:59:30.414124 12619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:59:30.415524 12619 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:59:30.417403 12619 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:59:30.417892 12619 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 14:59:30.418059 12619 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:59:30.418167 12619 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:59:30.499806 12619 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.2:40413
I20260928 14:59:30.499918 12759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.2:40413 every 8 connection(s)
I20260928 14:59:30.502007 12619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
I20260928 14:59:30.506738 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 12619
I20260928 14:59:30.507506 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.40.20:58001/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.3:33363
--local_ip_for_outbound_sockets=127.31.40.3
--tserver_master_addrs=127.31.40.62:46757
--webserver_port=37761
--webserver_interface=127.31.40.3
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab
--principal=kudu/127.31.40.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:35347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.40.20:58001/kms with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 14:59:30 dist-test-slave-j968 krb5kdc[11406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607570, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.2@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:59:30.557536 12764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:30.518715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:58585 (local address 127.31.40.62:46757)
0928 14:59:30.538503 (+ 19788us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:30.538510 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:30.538542 (+ 32us) server_negotiation.cc:407] Connection header received
0928 14:59:30.538628 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:30.538635 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:30.538740 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:30.538914 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:59:30.541354 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:30.542594 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:30.545327 (+ 2733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:30.546011 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:30.546112 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:30.550736 (+ 4624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:59:30.550741 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:59:30.550745 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:59:30.550781 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:59:30.553855 (+ 3074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:30.554671 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:30.554676 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:30.554678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:30.554802 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:30.555306 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:30.555310 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:30.555312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:30.555608 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:59:30.555774 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:30.556852 (+ 1078us) server_negotiation.cc:299] Negotiation successful
0928 14:59:30.557130 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19609,"thread_start_us":19481,"threads_started":1}
I20260928 14:59:30.560477 12763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:30.518079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:46757 (local address 127.31.40.2:58585)
0928 14:59:30.519025 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:30.519085 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 14:59:30.521646 (+ 2561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:59:30.539353 (+ 17707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:59:30.539370 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:59:30.540163 (+ 793us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:59:30.541089 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:30.541119 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:30.542836 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:30.542842 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:30.545126 (+ 2284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:30.545143 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:30.546620 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:30.546627 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:30.546932 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:30.547777 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:59:30.547815 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:59:30.550469 (+ 2654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:59:30.554020 (+ 3551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:59:30.554031 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:59:30.554051 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:59:30.554514 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:59:30.554918 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:59:30.554923 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:59:30.554927 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:59:30.555159 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:59:30.555749 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:59:30.555758 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:59:30.556332 (+ 574us) client_negotiation.cc:769] Sending connection context
0928 14:59:30.558192 (+ 1860us) client_negotiation.cc:240] Negotiation successful
0928 14:59:30.558889 (+ 697us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":666,"threads_started":1}
I20260928 14:59:30.562753 12760 heartbeater.cc:347] Connected to a master server at 127.31.40.62:46757
I20260928 14:59:30.563143 12760 heartbeater.cc:464] Registering TS with master...
I20260928 14:59:30.564177 12760 heartbeater.cc:511] Master 127.31.40.62:46757 requested a full tablet report, sending...
I20260928 14:59:30.566656 12417 ts_manager.cc:194] Registered new tserver with Master: 70933c12e76e4b5f900f69ccf6150213 (127.31.40.2:40413)
I20260928 14:59:30.569053 12417 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.2@KRBTEST.COM'} at 127.31.40.2:58585
Sep 28 14:59:30 dist-test-slave-j968 krb5kdc[11406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607570, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.62@KRBTEST.COM for HTTP/127.31.40.20@KRBTEST.COM
I20260928 14:59:30.831211 12615 heartbeater.cc:464] Registering TS with master...
I20260928 14:59:30.831688 12615 heartbeater.cc:511] Master 127.31.40.62:46757 requested a full tablet report, sending...
I20260928 14:59:30.833030 12417 ts_manager.cc:194] Registered new tserver with Master: 07672459d9cc4841aa75d13bbdfcfc6d (127.31.40.1:46251)
I20260928 14:59:30.834239 12417 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.1@KRBTEST.COM'} at 127.31.40.1:36357
W20260928 14:59:31.006683 12762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:59:31.007251 12762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:59:31.007465 12762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:59:31.019286 12762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:59:31.019538 12762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:59:31.019771 12762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.3
I20260928 14:59:31.035542 12762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:35347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.40.20:58001/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab
--principal=kudu/127.31.40.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.3:33363
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.40.3
--webserver_port=37761
--tserver_master_addrs=127.31.40.62:46757
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.3
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:59:31.037680 12762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:59:31.039399 12762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:59:31.079612 12770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:59:31.083559 12773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:59:31.084147 12762 server_base.cc:1034] running on GCE node
W20260928 14:59:31.084003 12769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:59:31.085284 12762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:59:31.092073 12762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:59:31.093353 12762 hybrid_clock.cc:648] HybridClock initialized: now 1790607571093276 us; error 76 us; skew 500 ppm
I20260928 14:59:31.093820 12762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:59:31 dist-test-slave-j968 krb5kdc[11406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607571, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:59:31.098079 12762 init.cc:377] Logged in from keytab as kudu/127.31.40.3@KRBTEST.COM (short username kudu)
I20260928 14:59:31.118746 12762 webserver.cc:460] Webserver started at http://127.31.40.3:37761/ using document root <none> and password file <none>
I20260928 14:59:31.119948 12762 fs_manager.cc:359] Metadata directory not provided
I20260928 14:59:31.120200 12762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 14:59:31 dist-test-slave-j968 krb5kdc[11406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607571, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.3@KRBTEST.COM for HTTP/127.31.40.20@KRBTEST.COM
I20260928 14:59:31.169134 12762 fs_manager.cc:709] Time spent opening directory manager: real 0.017s user 0.004s sys 0.002s
I20260928 14:59:31.179632 12781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
Sep 28 14:59:31 dist-test-slave-j968 krb5kdc[11406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607570, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.62@KRBTEST.COM for HTTP/127.31.40.20@KRBTEST.COM
I20260928 14:59:31.181931 12762 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.005s sys 0.005s
I20260928 14:59:31.182140 12762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-2/wal
uuid: "05bc129bf3744e87b1844b7447b39055"
format_stamp: "Formatted at 2026-09-28 14:59:27 on dist-test-slave-j968"
server_key: "98f6a999917a90b8b48af2771eaf9109"
server_key_iv: "99ec95b7f09068d10263016ad0ae6a61"
server_key_version: "kuduclusterkey@0"
I20260928 14:59:31.183910 12762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:59:31.291144 12762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:59:31.296533 12762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:59:31.297009 12762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:59:31.314694 12762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:59:31.323305 12762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:59:31.323531 12762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 14:59:31.323707 12762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:59:31.323822 12762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:59:31.436764 12762 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.3:33363
I20260928 14:59:31.436800 12893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.3:33363 every 8 connection(s)
I20260928 14:59:31.438338 12762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
I20260928 14:59:31.444253 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 12762
Sep 28 14:59:31 dist-test-slave-j968 krb5kdc[11406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607571, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.3@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:59:31.475467 12897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:31.442599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:46757 (local address 127.31.40.3:38379)
0928 14:59:31.452119 (+ 9520us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:31.452191 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0928 14:59:31.455328 (+ 3137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:59:31.456346 (+ 1018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:59:31.456359 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:59:31.456987 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:59:31.457847 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:31.457873 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:31.459198 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:31.459204 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:31.461317 (+ 2113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:31.461334 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:31.462613 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:31.462622 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:31.462921 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:31.463667 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:59:31.463698 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:59:31.466469 (+ 2771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:59:31.470385 (+ 3916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:59:31.470395 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:59:31.470414 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:59:31.471072 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:59:31.471529 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:59:31.471534 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:59:31.471538 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:59:31.471817 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:59:31.472590 (+ 773us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:59:31.472599 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:59:31.473084 (+ 485us) client_negotiation.cc:769] Sending connection context
0928 14:59:31.473405 (+ 321us) client_negotiation.cc:240] Negotiation successful
0928 14:59:31.474010 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9141,"thread_start_us":8879,"threads_started":1}
I20260928 14:59:31.476953 12896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:31.441721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:38379 (local address 127.31.40.62:46757)
0928 14:59:31.449961 (+ 8240us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:31.449967 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:31.453123 (+ 3156us) server_negotiation.cc:407] Connection header received
0928 14:59:31.455789 (+ 2666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:31.455793 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:31.455923 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:31.456047 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:59:31.458083 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:31.459006 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:31.461581 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:31.462355 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:31.462516 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:31.466757 (+ 4241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:59:31.466764 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:59:31.466768 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:59:31.466817 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:59:31.470209 (+ 3392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:31.471231 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:31.471236 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:31.471240 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:31.471390 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:31.471995 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:31.472000 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:31.472004 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:31.472365 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:59:31.472517 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:31.476305 (+ 3788us) server_negotiation.cc:299] Negotiation successful
0928 14:59:31.476569 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8105,"thread_start_us":7972,"threads_started":1}
I20260928 14:59:31.478219 12894 heartbeater.cc:347] Connected to a master server at 127.31.40.62:46757
I20260928 14:59:31.478574 12894 heartbeater.cc:464] Registering TS with master...
I20260928 14:59:31.479532 12894 heartbeater.cc:511] Master 127.31.40.62:46757 requested a full tablet report, sending...
I20260928 14:59:31.481256 12417 ts_manager.cc:194] Registered new tserver with Master: 05bc129bf3744e87b1844b7447b39055 (127.31.40.3:33363)
I20260928 14:59:31.481729 12899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:31.449968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48664 (local address 127.31.40.62:46757)
0928 14:59:31.453306 (+ 3338us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:31.453314 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:31.453333 (+ 19us) server_negotiation.cc:407] Connection header received
0928 14:59:31.453408 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:31.453416 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:31.453554 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:31.453706 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:59:31.455940 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:31.456783 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:31.458177 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:31.458893 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:31.459021 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:31.462030 (+ 3009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:59:31.462043 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:59:31.462047 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:59:31.462088 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:59:31.465453 (+ 3365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:31.466282 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:31.466288 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:31.466291 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:31.466461 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:31.478544 (+ 12083us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:31.478551 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:31.478554 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:31.478983 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:59:31.480742 (+ 1759us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:31.480914 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 14:59:31.481173 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3180,"thread_start_us":3016,"threads_started":1}
I20260928 14:59:31.482528 12417 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.3@KRBTEST.COM'} at 127.31.40.3:38379
I20260928 14:59:31.484946 31904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:59:31.507596 12896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:31.489322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48672 (local address 127.31.40.62:46757)
0928 14:59:31.489505 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:31.489512 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:31.490233 (+ 721us) server_negotiation.cc:407] Connection header received
0928 14:59:31.490380 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:31.490384 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:31.490470 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:31.490573 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:59:31.492599 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:31.493704 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:31.495434 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:31.496160 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:31.496297 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:31.500564 (+ 4267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:59:31.500572 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:59:31.500577 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:59:31.500623 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:59:31.504321 (+ 3698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:31.505349 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:31.505356 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:31.505360 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:31.505527 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:31.505996 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:31.506002 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:31.506006 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:31.506396 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:59:31.506646 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:31.506986 (+ 340us) server_negotiation.cc:299] Negotiation successful
0928 14:59:31.507221 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 14:59:31.513235 12417 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48672:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:59:31.521534 12417 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:59:31.557765 12921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:31.539145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51208 (local address 127.31.40.1:46251)
0928 14:59:31.549090 (+ 9945us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:31.549101 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:31.549135 (+ 34us) server_negotiation.cc:407] Connection header received
0928 14:59:31.549247 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:31.549256 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:31.549597 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:31.549828 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:59:31.550710 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:31.552486 (+ 1776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:31.555071 (+ 2585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:31.556557 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:31.556712 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:31.556984 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:31.557243 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 14:59:31.557410 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9720,"thread_start_us":9573,"threads_started":1}
I20260928 14:59:31.558130 12917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:31.538480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:46251 (local address 127.0.0.1:51208)
0928 14:59:31.548859 (+ 10379us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:31.548879 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 14:59:31.549022 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:59:31.549887 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:59:31.549897 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:59:31.549932 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:59:31.550415 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:31.550431 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:31.552627 (+ 2196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:31.552632 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:31.554853 (+ 2221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:31.554868 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:31.556768 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:31.556774 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:31.556979 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:31.557054 (+ 75us) client_negotiation.cc:769] Sending connection context
0928 14:59:31.557199 (+ 145us) client_negotiation.cc:240] Negotiation successful
0928 14:59:31.557410 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10252,"mutex_wait_us":73,"thread_start_us":8514,"threads_started":1}
I20260928 14:59:31.558504 12916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:31.536817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.3:33363 (local address 127.0.0.1:43910)
0928 14:59:31.537689 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:31.537738 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 14:59:31.537935 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:59:31.550355 (+ 12420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:59:31.550360 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:59:31.550396 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:59:31.550921 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:31.550934 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:31.552868 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:31.552874 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:31.555537 (+ 2663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:31.555554 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:31.557552 (+ 1998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:31.557559 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:31.557733 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:31.557769 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 14:59:31.557838 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 14:59:31.557980 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"mutex_wait_us":130,"thread_start_us":508,"threads_started":1}
I20260928 14:59:31.559072 12918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:31.537208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43910 (local address 127.31.40.3:33363)
0928 14:59:31.549759 (+ 12551us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:31.549770 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:31.549802 (+ 32us) server_negotiation.cc:407] Connection header received
0928 14:59:31.549897 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:31.549906 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:31.550133 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:31.550324 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:59:31.551149 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:31.552719 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:31.555745 (+ 3026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:31.557300 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:31.558237 (+ 937us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:31.558475 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:31.558545 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 14:59:31.558708 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12357,"thread_start_us":12212,"threads_started":1}
I20260928 14:59:31.559504 12920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:31.537044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.2:40413 (local address 127.0.0.1:41960)
0928 14:59:31.547364 (+ 10320us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:31.547397 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 14:59:31.549064 (+ 1667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:59:31.550133 (+ 1069us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:59:31.550139 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:59:31.550168 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:59:31.550813 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:31.550831 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:31.552948 (+ 2117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:31.552952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:31.556129 (+ 3177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:31.556144 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:31.558267 (+ 2123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:31.558272 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:31.558465 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:31.558502 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 14:59:31.558588 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 14:59:31.558710 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10129,"mutex_wait_us":133,"thread_start_us":12074,"threads_started":1}
I20260928 14:59:31.560750 12919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:31.537782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41960 (local address 127.31.40.2:40413)
0928 14:59:31.546913 (+ 9131us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:31.546925 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:31.549293 (+ 2368us) server_negotiation.cc:407] Connection header received
0928 14:59:31.549457 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:31.549465 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:31.549716 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:31.549943 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:59:31.551050 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:31.552734 (+ 1684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:31.556299 (+ 3565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:31.557769 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:31.557889 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:31.558109 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:31.559110 (+ 1001us) server_negotiation.cc:299] Negotiation successful
0928 14:59:31.559420 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8929,"thread_start_us":8775,"threads_started":1}
I20260928 14:59:31.562729 12829 tablet_service.cc:1467] Processing CreateTablet for tablet 653ce098e0744b9e9fe3b9fa91caa570 (DEFAULT_TABLE table=test-table [id=a34685bd987a4f89b0f96b1da9e6ba8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:59:31.562767 12550 tablet_service.cc:1467] Processing CreateTablet for tablet 653ce098e0744b9e9fe3b9fa91caa570 (DEFAULT_TABLE table=test-table [id=a34685bd987a4f89b0f96b1da9e6ba8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:59:31.564123 12829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 653ce098e0744b9e9fe3b9fa91caa570. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:59:31.564172 12550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 653ce098e0744b9e9fe3b9fa91caa570. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:59:31.564483 12695 tablet_service.cc:1467] Processing CreateTablet for tablet 653ce098e0744b9e9fe3b9fa91caa570 (DEFAULT_TABLE table=test-table [id=a34685bd987a4f89b0f96b1da9e6ba8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:59:31.565922 12695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 653ce098e0744b9e9fe3b9fa91caa570. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:59:31.574322 12760 heartbeater.cc:503] Master 127.31.40.62:46757 was elected leader, sending a full tablet report...
I20260928 14:59:31.577386 12922 tablet_bootstrap.cc:492] T 653ce098e0744b9e9fe3b9fa91caa570 P 05bc129bf3744e87b1844b7447b39055: Bootstrap starting.
I20260928 14:59:31.580755 12923 tablet_bootstrap.cc:492] T 653ce098e0744b9e9fe3b9fa91caa570 P 07672459d9cc4841aa75d13bbdfcfc6d: Bootstrap starting.
I20260928 14:59:31.582708 12922 tablet_bootstrap.cc:654] T 653ce098e0744b9e9fe3b9fa91caa570 P 05bc129bf3744e87b1844b7447b39055: Neither blocks nor log segments found. Creating new log.
I20260928 14:59:31.584266 12922 log.cc:824] T 653ce098e0744b9e9fe3b9fa91caa570 P 05bc129bf3744e87b1844b7447b39055: Log is configured to *not* fsync() on all Append() calls
I20260928 14:59:31.584460 12923 tablet_bootstrap.cc:654] T 653ce098e0744b9e9fe3b9fa91caa570 P 07672459d9cc4841aa75d13bbdfcfc6d: Neither blocks nor log segments found. Creating new log.
I20260928 14:59:31.584782 12924 tablet_bootstrap.cc:492] T 653ce098e0744b9e9fe3b9fa91caa570 P 70933c12e76e4b5f900f69ccf6150213: Bootstrap starting.
I20260928 14:59:31.586313 12923 log.cc:824] T 653ce098e0744b9e9fe3b9fa91caa570 P 07672459d9cc4841aa75d13bbdfcfc6d: Log is configured to *not* fsync() on all Append() calls
I20260928 14:59:31.588449 12924 tablet_bootstrap.cc:654] T 653ce098e0744b9e9fe3b9fa91caa570 P 70933c12e76e4b5f900f69ccf6150213: Neither blocks nor log segments found. Creating new log.
I20260928 14:59:31.589453 12924 log.cc:824] T 653ce098e0744b9e9fe3b9fa91caa570 P 70933c12e76e4b5f900f69ccf6150213: Log is configured to *not* fsync() on all Append() calls
I20260928 14:59:31.589564 12923 tablet_bootstrap.cc:492] T 653ce098e0744b9e9fe3b9fa91caa570 P 07672459d9cc4841aa75d13bbdfcfc6d: No bootstrap required, opened a new log
I20260928 14:59:31.589780 12923 ts_tablet_manager.cc:1397] T 653ce098e0744b9e9fe3b9fa91caa570 P 07672459d9cc4841aa75d13bbdfcfc6d: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260928 14:59:31.592744 12924 tablet_bootstrap.cc:492] T 653ce098e0744b9e9fe3b9fa91caa570 P 70933c12e76e4b5f900f69ccf6150213: No bootstrap required, opened a new log
I20260928 14:59:31.592795 12922 tablet_bootstrap.cc:492] T 653ce098e0744b9e9fe3b9fa91caa570 P 05bc129bf3744e87b1844b7447b39055: No bootstrap required, opened a new log
I20260928 14:59:31.593026 12924 ts_tablet_manager.cc:1397] T 653ce098e0744b9e9fe3b9fa91caa570 P 70933c12e76e4b5f900f69ccf6150213: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260928 14:59:31.593160 12922 ts_tablet_manager.cc:1397] T 653ce098e0744b9e9fe3b9fa91caa570 P 05bc129bf3744e87b1844b7447b39055: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.000s
I20260928 14:59:31.597689 12923 raft_consensus.cc:348] T 653ce098e0744b9e9fe3b9fa91caa570 P 07672459d9cc4841aa75d13bbdfcfc6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bc129bf3744e87b1844b7447b39055" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33363 } } peers { permanent_uuid: "70933c12e76e4b5f900f69ccf6150213" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 40413 } } peers { permanent_uuid: "07672459d9cc4841aa75d13bbdfcfc6d" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 46251 } }
I20260928 14:59:31.598120 12923 raft_consensus.cc:374] T 653ce098e0744b9e9fe3b9fa91caa570 P 07672459d9cc4841aa75d13bbdfcfc6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:59:31.598222 12923 raft_consensus.cc:729] T 653ce098e0744b9e9fe3b9fa91caa570 P 07672459d9cc4841aa75d13bbdfcfc6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07672459d9cc4841aa75d13bbdfcfc6d, State: Initialized, Role: FOLLOWER
I20260928 14:59:31.598881 12923 consensus_queue.cc:260] T 653ce098e0744b9e9fe3b9fa91caa570 P 07672459d9cc4841aa75d13bbdfcfc6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bc129bf3744e87b1844b7447b39055" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33363 } } peers { permanent_uuid: "70933c12e76e4b5f900f69ccf6150213" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 40413 } } peers { permanent_uuid: "07672459d9cc4841aa75d13bbdfcfc6d" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 46251 } }
I20260928 14:59:31.600602 12615 heartbeater.cc:503] Master 127.31.40.62:46757 was elected leader, sending a full tablet report...
I20260928 14:59:31.601321 12923 ts_tablet_manager.cc:1428] T 653ce098e0744b9e9fe3b9fa91caa570 P 07672459d9cc4841aa75d13bbdfcfc6d: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260928 14:59:31.603960 12924 raft_consensus.cc:348] T 653ce098e0744b9e9fe3b9fa91caa570 P 70933c12e76e4b5f900f69ccf6150213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bc129bf3744e87b1844b7447b39055" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33363 } } peers { permanent_uuid: "70933c12e76e4b5f900f69ccf6150213" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 40413 } } peers { permanent_uuid: "07672459d9cc4841aa75d13bbdfcfc6d" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 46251 } }
I20260928 14:59:31.603957 12922 raft_consensus.cc:348] T 653ce098e0744b9e9fe3b9fa91caa570 P 05bc129bf3744e87b1844b7447b39055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bc129bf3744e87b1844b7447b39055" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33363 } } peers { permanent_uuid: "70933c12e76e4b5f900f69ccf6150213" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 40413 } } peers { permanent_uuid: "07672459d9cc4841aa75d13bbdfcfc6d" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 46251 } }
I20260928 14:59:31.604410 12924 raft_consensus.cc:374] T 653ce098e0744b9e9fe3b9fa91caa570 P 70933c12e76e4b5f900f69ccf6150213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:59:31.604409 12922 raft_consensus.cc:374] T 653ce098e0744b9e9fe3b9fa91caa570 P 05bc129bf3744e87b1844b7447b39055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:59:31.604511 12922 raft_consensus.cc:729] T 653ce098e0744b9e9fe3b9fa91caa570 P 05bc129bf3744e87b1844b7447b39055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05bc129bf3744e87b1844b7447b39055, State: Initialized, Role: FOLLOWER
I20260928 14:59:31.604511 12924 raft_consensus.cc:729] T 653ce098e0744b9e9fe3b9fa91caa570 P 70933c12e76e4b5f900f69ccf6150213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70933c12e76e4b5f900f69ccf6150213, State: Initialized, Role: FOLLOWER
I20260928 14:59:31.605162 12922 consensus_queue.cc:260] T 653ce098e0744b9e9fe3b9fa91caa570 P 05bc129bf3744e87b1844b7447b39055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bc129bf3744e87b1844b7447b39055" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33363 } } peers { permanent_uuid: "70933c12e76e4b5f900f69ccf6150213" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 40413 } } peers { permanent_uuid: "07672459d9cc4841aa75d13bbdfcfc6d" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 46251 } }
I20260928 14:59:31.605162 12924 consensus_queue.cc:260] T 653ce098e0744b9e9fe3b9fa91caa570 P 70933c12e76e4b5f900f69ccf6150213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bc129bf3744e87b1844b7447b39055" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33363 } } peers { permanent_uuid: "70933c12e76e4b5f900f69ccf6150213" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 40413 } } peers { permanent_uuid: "07672459d9cc4841aa75d13bbdfcfc6d" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 46251 } }
I20260928 14:59:31.607183 12922 ts_tablet_manager.cc:1428] T 653ce098e0744b9e9fe3b9fa91caa570 P 05bc129bf3744e87b1844b7447b39055: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260928 14:59:31.607290 12924 ts_tablet_manager.cc:1428] T 653ce098e0744b9e9fe3b9fa91caa570 P 70933c12e76e4b5f900f69ccf6150213: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260928 14:59:31.607807 12894 heartbeater.cc:503] Master 127.31.40.62:46757 was elected leader, sending a full tablet report...
I20260928 14:59:31.663656 12929 raft_consensus.cc:482] T 653ce098e0744b9e9fe3b9fa91caa570 P 70933c12e76e4b5f900f69ccf6150213 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:59:31.664014 12929 raft_consensus.cc:504] T 653ce098e0744b9e9fe3b9fa91caa570 P 70933c12e76e4b5f900f69ccf6150213 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bc129bf3744e87b1844b7447b39055" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33363 } } peers { permanent_uuid: "70933c12e76e4b5f900f69ccf6150213" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 40413 } } peers { permanent_uuid: "07672459d9cc4841aa75d13bbdfcfc6d" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 46251 } }
I20260928 14:59:31.665979 12929 leader_election.cc:288] T 653ce098e0744b9e9fe3b9fa91caa570 P 70933c12e76e4b5f900f69ccf6150213 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05bc129bf3744e87b1844b7447b39055 (127.31.40.3:33363), 07672459d9cc4841aa75d13bbdfcfc6d (127.31.40.1:46251)
I20260928 14:59:31.673723 12918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:31.666307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:54037 (local address 127.31.40.3:33363)
0928 14:59:31.666590 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:31.666598 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:31.666974 (+ 376us) server_negotiation.cc:407] Connection header received
0928 14:59:31.667270 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:31.667278 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:31.667357 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:31.667424 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:59:31.668451 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:31.669619 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:31.671559 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:31.672695 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:31.672823 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:31.672931 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:31.673338 (+ 407us) server_negotiation.cc:299] Negotiation successful
0928 14:59:31.673466 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 14:59:31.673897 12921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:31.667088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:56073 (local address 127.31.40.1:46251)
0928 14:59:31.667302 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:31.667309 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:31.667550 (+ 241us) server_negotiation.cc:407] Connection header received
0928 14:59:31.667706 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:31.667710 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:31.667780 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:31.667857 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:59:31.668578 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:31.669651 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:31.671520 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:31.672716 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:31.672852 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:31.672969 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:31.673503 (+ 534us) server_negotiation.cc:299] Negotiation successful
0928 14:59:31.673652 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 14:59:31.674216 12931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:31.666176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.3:33363 (local address 127.31.40.2:54037)
0928 14:59:31.666867 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:31.666887 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 14:59:31.667068 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:59:31.667645 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:59:31.667652 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:59:31.667679 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:59:31.668286 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:31.668305 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:31.669779 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:31.669783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:31.671356 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:31.671368 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:31.672887 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:31.672894 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:31.673102 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:31.673149 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 14:59:31.673308 (+ 159us) client_negotiation.cc:240] Negotiation successful
0928 14:59:31.673456 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"mutex_wait_us":60,"thread_start_us":336,"threads_started":1}
I20260928 14:59:31.674257 12932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:31.666851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:46251 (local address 127.31.40.2:56073)
0928 14:59:31.667453 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:31.667470 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:59:31.667591 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:59:31.667923 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:59:31.667928 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:59:31.667949 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:59:31.668429 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:31.668442 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:31.669822 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:31.669830 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:31.671360 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:31.671373 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:31.673114 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:31.673127 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:31.673323 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:31.673361 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 14:59:31.673456 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 14:59:31.673585 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"mutex_wait_us":81,"thread_start_us":286,"threads_started":1}
I20260928 14:59:31.675129 12849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "653ce098e0744b9e9fe3b9fa91caa570" candidate_uuid: "70933c12e76e4b5f900f69ccf6150213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05bc129bf3744e87b1844b7447b39055" is_pre_election: true
I20260928 14:59:31.675154 12570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "653ce098e0744b9e9fe3b9fa91caa570" candidate_uuid: "70933c12e76e4b5f900f69ccf6150213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07672459d9cc4841aa75d13bbdfcfc6d" is_pre_election: true
I20260928 14:59:31.675642 12849 raft_consensus.cc:2454] T 653ce098e0744b9e9fe3b9fa91caa570 P 05bc129bf3744e87b1844b7447b39055 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70933c12e76e4b5f900f69ccf6150213 in term 0.
I20260928 14:59:31.675642 12570 raft_consensus.cc:2454] T 653ce098e0744b9e9fe3b9fa91caa570 P 07672459d9cc4841aa75d13bbdfcfc6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70933c12e76e4b5f900f69ccf6150213 in term 0.
I20260928 14:59:31.676306 12648 leader_election.cc:302] T 653ce098e0744b9e9fe3b9fa91caa570 P 70933c12e76e4b5f900f69ccf6150213 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05bc129bf3744e87b1844b7447b39055, 70933c12e76e4b5f900f69ccf6150213; no voters:
I20260928 14:59:31.676693 12929 raft_consensus.cc:2781] T 653ce098e0744b9e9fe3b9fa91caa570 P 70933c12e76e4b5f900f69ccf6150213 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:59:31.676846 12929 raft_consensus.cc:482] T 653ce098e0744b9e9fe3b9fa91caa570 P 70933c12e76e4b5f900f69ccf6150213 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:59:31.677004 12929 raft_consensus.cc:3037] T 653ce098e0744b9e9fe3b9fa91caa570 P 70933c12e76e4b5f900f69ccf6150213 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:59:31.679390 12929 raft_consensus.cc:504] T 653ce098e0744b9e9fe3b9fa91caa570 P 70933c12e76e4b5f900f69ccf6150213 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bc129bf3744e87b1844b7447b39055" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33363 } } peers { permanent_uuid: "70933c12e76e4b5f900f69ccf6150213" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 40413 } } peers { permanent_uuid: "07672459d9cc4841aa75d13bbdfcfc6d" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 46251 } }
I20260928 14:59:31.680150 12929 leader_election.cc:288] T 653ce098e0744b9e9fe3b9fa91caa570 P 70933c12e76e4b5f900f69ccf6150213 [CANDIDATE]: Term 1 election: Requested vote from peers 05bc129bf3744e87b1844b7447b39055 (127.31.40.3:33363), 07672459d9cc4841aa75d13bbdfcfc6d (127.31.40.1:46251)
I20260928 14:59:31.680677 12849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "653ce098e0744b9e9fe3b9fa91caa570" candidate_uuid: "70933c12e76e4b5f900f69ccf6150213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05bc129bf3744e87b1844b7447b39055"
I20260928 14:59:31.680714 12570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "653ce098e0744b9e9fe3b9fa91caa570" candidate_uuid: "70933c12e76e4b5f900f69ccf6150213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07672459d9cc4841aa75d13bbdfcfc6d"
I20260928 14:59:31.680915 12849 raft_consensus.cc:3037] T 653ce098e0744b9e9fe3b9fa91caa570 P 05bc129bf3744e87b1844b7447b39055 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:59:31.680915 12570 raft_consensus.cc:3037] T 653ce098e0744b9e9fe3b9fa91caa570 P 07672459d9cc4841aa75d13bbdfcfc6d [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:59:31.684194 12849 raft_consensus.cc:2454] T 653ce098e0744b9e9fe3b9fa91caa570 P 05bc129bf3744e87b1844b7447b39055 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70933c12e76e4b5f900f69ccf6150213 in term 1.
I20260928 14:59:31.684194 12570 raft_consensus.cc:2454] T 653ce098e0744b9e9fe3b9fa91caa570 P 07672459d9cc4841aa75d13bbdfcfc6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70933c12e76e4b5f900f69ccf6150213 in term 1.
I20260928 14:59:31.684649 12648 leader_election.cc:302] T 653ce098e0744b9e9fe3b9fa91caa570 P 70933c12e76e4b5f900f69ccf6150213 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07672459d9cc4841aa75d13bbdfcfc6d, 70933c12e76e4b5f900f69ccf6150213; no voters:
I20260928 14:59:31.684993 12929 raft_consensus.cc:2781] T 653ce098e0744b9e9fe3b9fa91caa570 P 70933c12e76e4b5f900f69ccf6150213 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:59:31.685551 12929 raft_consensus.cc:686] T 653ce098e0744b9e9fe3b9fa91caa570 P 70933c12e76e4b5f900f69ccf6150213 [term 1 LEADER]: Becoming Leader. State: Replica: 70933c12e76e4b5f900f69ccf6150213, State: Running, Role: LEADER
I20260928 14:59:31.686142 12929 consensus_queue.cc:237] T 653ce098e0744b9e9fe3b9fa91caa570 P 70933c12e76e4b5f900f69ccf6150213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bc129bf3744e87b1844b7447b39055" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33363 } } peers { permanent_uuid: "70933c12e76e4b5f900f69ccf6150213" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 40413 } } peers { permanent_uuid: "07672459d9cc4841aa75d13bbdfcfc6d" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 46251 } }
I20260928 14:59:31.690795 12417 catalog_manager.cc:5495] T 653ce098e0744b9e9fe3b9fa91caa570 P 70933c12e76e4b5f900f69ccf6150213 reported cstate change: term changed from 0 to 1, leader changed from <none> to 70933c12e76e4b5f900f69ccf6150213 (127.31.40.2). New cstate: current_term: 1 leader_uuid: "70933c12e76e4b5f900f69ccf6150213" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bc129bf3744e87b1844b7447b39055" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70933c12e76e4b5f900f69ccf6150213" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 40413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07672459d9cc4841aa75d13bbdfcfc6d" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 46251 } health_report { overall_health: UNKNOWN } } }
W20260928 14:59:31.704613 12895 tablet.cc:2369] T 653ce098e0744b9e9fe3b9fa91caa570 P 05bc129bf3744e87b1844b7447b39055: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:59:31.715394 12919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:31.710148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41970 (local address 127.31.40.2:40413)
0928 14:59:31.710331 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:31.710335 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:31.710353 (+ 18us) server_negotiation.cc:407] Connection header received
0928 14:59:31.710484 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:31.710487 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:31.710581 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:31.710700 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:59:31.711102 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:31.711816 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:31.713258 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:31.713658 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:31.713748 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:31.714241 (+ 493us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:59:31.714788 (+ 547us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:59:31.714897 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:31.715017 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 14:59:31.715082 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 14:59:31.724133 12570 raft_consensus.cc:1261] T 653ce098e0744b9e9fe3b9fa91caa570 P 07672459d9cc4841aa75d13bbdfcfc6d [term 1 FOLLOWER]: Refusing update from remote peer 70933c12e76e4b5f900f69ccf6150213: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:59:31.724131 12849 raft_consensus.cc:1261] T 653ce098e0744b9e9fe3b9fa91caa570 P 05bc129bf3744e87b1844b7447b39055 [term 1 FOLLOWER]: Refusing update from remote peer 70933c12e76e4b5f900f69ccf6150213: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:59:31.725186 12933 consensus_queue.cc:1035] T 653ce098e0744b9e9fe3b9fa91caa570 P 70933c12e76e4b5f900f69ccf6150213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07672459d9cc4841aa75d13bbdfcfc6d" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 46251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:59:31.725560 12929 consensus_queue.cc:1035] T 653ce098e0744b9e9fe3b9fa91caa570 P 70933c12e76e4b5f900f69ccf6150213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05bc129bf3744e87b1844b7447b39055" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 33363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:59:31.737739 12936 mvcc.cc:205] Tried to move back new op lower bound from 7334328613771243520 to 7334328613630685184. Current Snapshot: MvccSnapshot[applied={T|T < 7334328613771243520}]
I20260928 14:59:31.752285 12416 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:59:31.752568 12416 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:59:31.756506 12416 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e82e9783177b407fba3ef6b25c48b2cc: Sending DeleteTablet for 3 replicas of tablet 653ce098e0744b9e9fe3b9fa91caa570
I20260928 14:59:31.757750 12695 tablet_service.cc:1514] Processing DeleteTablet for tablet 653ce098e0744b9e9fe3b9fa91caa570 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:59:31 UTC) from {username='kudu', principal='kudu/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:41960
I20260928 14:59:31.758100 12550 tablet_service.cc:1514] Processing DeleteTablet for tablet 653ce098e0744b9e9fe3b9fa91caa570 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:59:31 UTC) from {username='kudu', principal='kudu/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:51208
I20260928 14:59:31.758148 12829 tablet_service.cc:1514] Processing DeleteTablet for tablet 653ce098e0744b9e9fe3b9fa91caa570 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:59:31 UTC) from {username='kudu', principal='kudu/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:43910
I20260928 14:59:31.760546 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 12454
I20260928 14:59:31.773988 12948 tablet_replica.cc:330] T 653ce098e0744b9e9fe3b9fa91caa570 P 05bc129bf3744e87b1844b7447b39055: stopping tablet replica
I20260928 14:59:31.774822 12948 raft_consensus.cc:2229] T 653ce098e0744b9e9fe3b9fa91caa570 P 05bc129bf3744e87b1844b7447b39055 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:59:31.775480 12948 raft_consensus.cc:2258] T 653ce098e0744b9e9fe3b9fa91caa570 P 05bc129bf3744e87b1844b7447b39055 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:59:31.776520 12947 tablet_replica.cc:330] T 653ce098e0744b9e9fe3b9fa91caa570 P 07672459d9cc4841aa75d13bbdfcfc6d: stopping tablet replica
I20260928 14:59:31.777024 12947 raft_consensus.cc:2229] T 653ce098e0744b9e9fe3b9fa91caa570 P 07672459d9cc4841aa75d13bbdfcfc6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:59:31.777534 12947 raft_consensus.cc:2258] T 653ce098e0744b9e9fe3b9fa91caa570 P 07672459d9cc4841aa75d13bbdfcfc6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:59:31.778486 12946 tablet_replica.cc:330] T 653ce098e0744b9e9fe3b9fa91caa570 P 70933c12e76e4b5f900f69ccf6150213: stopping tablet replica
I20260928 14:59:31.779544 12948 ts_tablet_manager.cc:1905] T 653ce098e0744b9e9fe3b9fa91caa570 P 05bc129bf3744e87b1844b7447b39055: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:59:31.780529 12947 ts_tablet_manager.cc:1905] T 653ce098e0744b9e9fe3b9fa91caa570 P 07672459d9cc4841aa75d13bbdfcfc6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:59:31.784909 12946 raft_consensus.cc:2229] T 653ce098e0744b9e9fe3b9fa91caa570 P 70933c12e76e4b5f900f69ccf6150213 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:59:31.785948 12921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:31.761739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51218 (local address 127.31.40.1:46251)
0928 14:59:31.762017 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:31.762023 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:31.762081 (+ 58us) server_negotiation.cc:407] Connection header received
0928 14:59:31.762343 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:31.762351 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:31.762463 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:31.762577 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:59:31.764766 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:31.765656 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:31.768557 (+ 2901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:31.769545 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:31.769721 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:31.776348 (+ 6627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:59:31.776362 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:59:31.776382 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:59:31.776435 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:59:31.780680 (+ 4245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:31.781727 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:31.781736 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:31.781740 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:31.781943 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:31.783808 (+ 1865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:31.783814 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:31.783818 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:31.784317 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:59:31.784653 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:31.785070 (+ 417us) server_negotiation.cc:299] Negotiation successful
0928 14:59:31.785356 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 14:59:31.786798 12610 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:59:31.787096 12946 raft_consensus.cc:2258] T 653ce098e0744b9e9fe3b9fa91caa570 P 70933c12e76e4b5f900f69ccf6150213 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:59:31.791286 12946 ts_tablet_manager.cc:1905] T 653ce098e0744b9e9fe3b9fa91caa570 P 70933c12e76e4b5f900f69ccf6150213: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:59:31.793147 12948 ts_tablet_manager.cc:1918] T 653ce098e0744b9e9fe3b9fa91caa570 P 05bc129bf3744e87b1844b7447b39055: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:59:31.793442 12948 log.cc:1196] T 653ce098e0744b9e9fe3b9fa91caa570 P 05bc129bf3744e87b1844b7447b39055: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-2/wal/wals/653ce098e0744b9e9fe3b9fa91caa570
I20260928 14:59:31.794194 12948 ts_tablet_manager.cc:1939] T 653ce098e0744b9e9fe3b9fa91caa570 P 05bc129bf3744e87b1844b7447b39055: Deleting consensus metadata
I20260928 14:59:31.796912 12401 catalog_manager.cc:4841] TS 05bc129bf3744e87b1844b7447b39055 (127.31.40.3:33363): tablet 653ce098e0744b9e9fe3b9fa91caa570 (table test-table [id=a34685bd987a4f89b0f96b1da9e6ba8f]) successfully deleted
I20260928 14:59:31.799019 12946 ts_tablet_manager.cc:1918] T 653ce098e0744b9e9fe3b9fa91caa570 P 70933c12e76e4b5f900f69ccf6150213: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:59:31.799306 12946 log.cc:1196] T 653ce098e0744b9e9fe3b9fa91caa570 P 70933c12e76e4b5f900f69ccf6150213: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-1/wal/wals/653ce098e0744b9e9fe3b9fa91caa570
I20260928 14:59:31.799970 12946 ts_tablet_manager.cc:1939] T 653ce098e0744b9e9fe3b9fa91caa570 P 70933c12e76e4b5f900f69ccf6150213: Deleting consensus metadata
I20260928 14:59:31.802038 12404 catalog_manager.cc:4841] TS 70933c12e76e4b5f900f69ccf6150213 (127.31.40.2:40413): tablet 653ce098e0744b9e9fe3b9fa91caa570 (table test-table [id=a34685bd987a4f89b0f96b1da9e6ba8f]) successfully deleted
I20260928 14:59:31.890709 12947 ts_tablet_manager.cc:1918] T 653ce098e0744b9e9fe3b9fa91caa570 P 07672459d9cc4841aa75d13bbdfcfc6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:59:31.890935 12947 log.cc:1196] T 653ce098e0744b9e9fe3b9fa91caa570 P 07672459d9cc4841aa75d13bbdfcfc6d: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607365362981-31904-0/minicluster-data/ts-0/wal/wals/653ce098e0744b9e9fe3b9fa91caa570
I20260928 14:59:31.891510 12947 ts_tablet_manager.cc:1939] T 653ce098e0744b9e9fe3b9fa91caa570 P 07672459d9cc4841aa75d13bbdfcfc6d: Deleting consensus metadata
I20260928 14:59:31.892297 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 12454
W20260928 14:59:31.916334 12401 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.40.1:46251 (error 108)
I20260928 14:59:31.916647 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 12619
I20260928 14:59:31.918870 12920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:31.918391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:46251 (local address 127.0.0.1:51226)
0928 14:59:31.918641 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:31.918733 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.1:46251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
W20260928 14:59:31.919162 12401 catalog_manager.cc:4584] TS 07672459d9cc4841aa75d13bbdfcfc6d (127.31.40.1:46251): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 653ce098e0744b9e9fe3b9fa91caa570: Network error: Client connection negotiation failed: client connection to 127.31.40.1:46251: connect: Connection refused (error 111)
I20260928 14:59:31.933593 12919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:31.917724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41980 (local address 127.31.40.2:40413)
0928 14:59:31.917915 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:31.917923 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:31.917947 (+ 24us) server_negotiation.cc:407] Connection header received
0928 14:59:31.918168 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:31.918173 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:31.918292 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:31.918403 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:59:31.920205 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:31.921031 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:31.923086 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:31.923543 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:31.923644 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:31.926998 (+ 3354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:59:31.927013 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:59:31.927031 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:59:31.927073 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:59:31.930375 (+ 3302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:31.931262 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:31.931269 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:31.931272 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:31.931417 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:31.931822 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:31.931825 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:31.931827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:31.932362 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:59:31.932698 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:31.932985 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 14:59:31.933231 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 14:59:31.940881 12755 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:59:31.972949 12920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:31.972357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:46251 (local address 127.0.0.1:51242)
0928 14:59:31.972599 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:31.972723 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.1:46251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260928 14:59:31.974066 12920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:31.973726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:46251 (local address 127.0.0.1:51246)
0928 14:59:31.973863 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:31.973961 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.1:46251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 14:59:32.034857 12920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:32.034356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:46251 (local address 127.0.0.1:51262)
0928 14:59:32.034586 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:32.034680 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.1:46251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260928 14:59:32.036069 12920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:32.035626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:46251 (local address 127.0.0.1:51278)
0928 14:59:32.035810 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:32.035933 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.1:46251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 14:59:32.059438 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 12619
I20260928 14:59:32.079041 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 12762
I20260928 14:59:32.097002 12918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:32.082209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43926 (local address 127.31.40.3:33363)
0928 14:59:32.082428 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:32.082435 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:32.082461 (+ 26us) server_negotiation.cc:407] Connection header received
0928 14:59:32.082772 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:32.082779 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:32.082895 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:32.083013 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:59:32.084401 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:32.085571 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:32.087254 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:32.087817 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:32.087988 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:32.090686 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:59:32.090700 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:59:32.090716 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:59:32.090845 (+ 129us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:59:32.094003 (+ 3158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:32.094698 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:32.094705 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:32.094708 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:32.094895 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:32.095390 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:32.095394 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:32.095396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:32.095769 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:59:32.096144 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:32.096438 (+ 294us) server_negotiation.cc:299] Negotiation successful
0928 14:59:32.096610 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 14:59:32.097684 12889 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:59:32.117262 12920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:32.116802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:46251 (local address 127.0.0.1:51284)
0928 14:59:32.117018 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:32.117108 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.1:46251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260928 14:59:32.118383 12920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:32.118032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:46251 (local address 127.0.0.1:51296)
0928 14:59:32.118215 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:32.118288 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.1:46251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260928 14:59:32.203433 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 12762
I20260928 14:59:32.221096 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 12384
I20260928 14:59:32.222443 12447 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:59:32.339323 12920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:32.338493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:46251 (local address 127.0.0.1:51300)
0928 14:59:32.338985 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:32.339096 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.1:46251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315}
I20260928 14:59:32.339371 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 12384
I20260928 14:59:32.365037 31904 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 14:59:32.700903 31904 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 14:59:32.701150 31904 mini_ranger.cc:66] Stopping Ranger...
I20260928 14:59:33.047466 31904 mini_ranger.cc:68] Stopped Ranger
2026-09-28 14:59:33.047 UTC [11436] LOG: received smart shutdown request
2026-09-28 14:59:33.051 UTC [11436] LOG: background worker "logical replication launcher" (PID 11445) exited with exit code 1
2026-09-28 14:59:33.051 UTC [11440] LOG: shutting down
2026-09-28 14:59:33.240 UTC [11436] LOG: database system is shut down
2026-09-28T14:59:33Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53740 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:59:33 dist-test-slave-j968 krb5kdc[12962](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:59:33 dist-test-slave-j968 krb5kdc[12962](info): set up 2 sockets
Sep 28 14:59:33 dist-test-slave-j968 krb5kdc[12962](info): commencing operation
krb5kdc: starting...
W20260928 14:59:34.477115 31904 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Sep 28 14:59:34 dist-test-slave-j968 krb5kdc[12962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:59:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:59:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab.
Entry for principal kudu/127.31.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab.
I20260928 14:59:34.727480 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:36353
--webserver_interface=127.31.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:33863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.40.62:36353,127.31.40.61:38411,127.31.40.60:38831
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:59:35.087507 12975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:59:35.087822 12975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:59:35.088055 12975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:59:35.095536 12975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:59:35.095649 12975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:59:35.095742 12975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:59:35.095851 12975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:59:35.095957 12975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:59:35.106029 12975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:33863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.40.62:36353,127.31.40.61:38411,127.31.40.60:38831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:36353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:59:35.107453 12975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:59:35.108860 12975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:59:35.121706 12983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:59:35.121734 12981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:59:35.121848 12980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:59:35.122452 12975 server_base.cc:1034] running on GCE node
I20260928 14:59:35.123380 12975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:59:35.124953 12975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:59:35.126191 12975 hybrid_clock.cc:648] HybridClock initialized: now 1790607575126142 us; error 44 us; skew 500 ppm
I20260928 14:59:35.126561 12975 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:59:35 dist-test-slave-j968 krb5kdc[12962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607575, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:59:35.137802 12975 init.cc:377] Logged in from keytab as kudu/127.31.40.62@KRBTEST.COM (short username kudu)
I20260928 14:59:35.139497 12975 webserver.cc:460] Webserver started at http://127.31.40.62:45267/ using document root <none> and password file <none>
I20260928 14:59:35.140336 12975 fs_manager.cc:359] Metadata directory not provided
I20260928 14:59:35.140492 12975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:59:35.140810 12975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:59:35.144114 12975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/minicluster-data/master-0/data/instance:
uuid: "b8e23f386b24452bb15a4efcdda75fea"
format_stamp: "Formatted at 2026-09-28 14:59:35 on dist-test-slave-j968"
server_key: "8bce85f9063d623998d64300c92e6288"
server_key_iv: "4f24406fe2fbd7c473b2d36f8b485d8c"
server_key_version: "encryptionkey@0"
I20260928 14:59:35.144996 12975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/minicluster-data/master-0/wal/instance:
uuid: "b8e23f386b24452bb15a4efcdda75fea"
format_stamp: "Formatted at 2026-09-28 14:59:35 on dist-test-slave-j968"
server_key: "8bce85f9063d623998d64300c92e6288"
server_key_iv: "4f24406fe2fbd7c473b2d36f8b485d8c"
server_key_version: "encryptionkey@0"
I20260928 14:59:35.150662 12975 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 14:59:35.154356 12990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:59:35.155857 12975 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 14:59:35.156131 12975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/minicluster-data/master-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/minicluster-data/master-0/wal
uuid: "b8e23f386b24452bb15a4efcdda75fea"
format_stamp: "Formatted at 2026-09-28 14:59:35 on dist-test-slave-j968"
server_key: "8bce85f9063d623998d64300c92e6288"
server_key_iv: "4f24406fe2fbd7c473b2d36f8b485d8c"
server_key_version: "encryptionkey@0"
I20260928 14:59:35.156347 12975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:59:35.172501 12975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:59:35.177074 12975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:59:35.177368 12975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:59:35.198933 12975 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.62:36353
I20260928 14:59:35.198994 13041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.62:36353 every 8 connection(s)
I20260928 14:59:35.200438 12975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
I20260928 14:59:35.202973 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 12975
I20260928 14:59:35.203253 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/minicluster-data/master-0/wal/instance
I20260928 14:59:35.204213 31904 external_mini_cluster.cc:1428] Setting key a1e4afd32c174813b2fc692ae30448a2
I20260928 14:59:35.205404 13042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:59:35.216112 13042 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.40.62" port: 36353 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.31.40.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.61@KRBTEST.COM" created.
Sep 28 14:59:35 dist-test-slave-j968 krb5kdc[12962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607575, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.62@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:59:35.244058 13046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:35.219941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:36353 (local address 127.0.0.1:49982)
0928 14:59:35.221292 (+ 1351us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:35.221351 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 14:59:35.224335 (+ 2984us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:59:35.225492 (+ 1157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:59:35.225502 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:59:35.226432 (+ 930us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:59:35.227039 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:35.227072 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:35.228787 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:35.228793 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:35.230189 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:35.230203 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:35.231211 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:35.231216 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:35.231337 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:35.232155 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:59:35.232188 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:59:35.234446 (+ 2258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:59:35.239075 (+ 4629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:59:35.239091 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:59:35.239107 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:59:35.239704 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:59:35.240242 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:59:35.240247 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:59:35.240251 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:59:35.240405 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:59:35.241263 (+ 858us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:59:35.241277 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:59:35.241531 (+ 254us) client_negotiation.cc:769] Sending connection context
0928 14:59:35.241814 (+ 283us) client_negotiation.cc:240] Negotiation successful
0928 14:59:35.242538 (+ 724us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1044,"mutex_wait_us":221,"spinlock_wait_cycles":70912,"thread_start_us":814,"threads_started":1}
I20260928 14:59:35.244082 13045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:35.219941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49982 (local address 127.31.40.62:36353)
0928 14:59:35.220970 (+ 1029us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:35.221001 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:35.221503 (+ 502us) server_negotiation.cc:407] Connection header received
0928 14:59:35.224875 (+ 3372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:35.224886 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:35.225208 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:35.225389 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:59:35.227289 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:35.228626 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:35.230320 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:35.230915 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:35.231171 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:35.234703 (+ 3532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:59:35.234717 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:59:35.234738 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:59:35.234785 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:59:35.238890 (+ 4105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:35.239892 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:35.239904 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:35.239909 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:35.240110 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:35.240521 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:35.240529 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:35.240532 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:35.241106 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:59:35.241327 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:35.241926 (+ 599us) server_negotiation.cc:299] Negotiation successful
0928 14:59:35.242422 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":733,"thread_start_us":469,"threads_started":1}
I20260928 14:59:35.247450 13042 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.40.61" port: 38411 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:59:35.248800 13046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:35.248337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.61:38411 (local address 127.0.0.1:37088)
0928 14:59:35.248553 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:35.248670 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.61:38411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260928 14:59:35.249150 12993 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.40.61:38411: connect: Connection refused (error 111)
I20260928 14:59:35.250918 13046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:35.250519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.61:38411 (local address 127.0.0.1:37100)
0928 14:59:35.250693 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:35.250769 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.61:38411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 14:59:35.251152 13042 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.40.61:38411: Network error: Client connection negotiation failed: client connection to 127.31.40.61:38411: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.61.keytab.
Entry for principal kudu/127.31.40.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.61.keytab.
I20260928 14:59:35.263082 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.61:38411
--webserver_interface=127.31.40.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.61.keytab
--principal=kudu/127.31.40.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:33863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.40.62:36353,127.31.40.61:38411,127.31.40.60:38831
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:59:35.300607 13042 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.40.61" port: 38411 } attempt: 1
I20260928 14:59:35.301960 13046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:35.301361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.61:38411 (local address 127.0.0.1:37110)
0928 14:59:35.301610 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:35.301742 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.61:38411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260928 14:59:35.303483 13046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:35.302942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.61:38411 (local address 127.0.0.1:37122)
0928 14:59:35.303206 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:35.303300 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.61:38411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
W20260928 14:59:35.303725 13042 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.40.61:38411: Network error: Client connection negotiation failed: client connection to 127.31.40.61:38411: connect: Connection refused (error 111)
I20260928 14:59:35.372002 13042 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.40.61" port: 38411 } attempt: 2
I20260928 14:59:35.373323 13046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:35.372754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.61:38411 (local address 127.0.0.1:37130)
0928 14:59:35.373077 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:35.373176 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.61:38411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
I20260928 14:59:35.374543 13046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:35.374143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.61:38411 (local address 127.0.0.1:37146)
0928 14:59:35.374327 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:35.374405 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.61:38411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 14:59:35.374761 13042 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.40.61:38411: Network error: Client connection negotiation failed: client connection to 127.31.40.61:38411: connect: Connection refused (error 111)
I20260928 14:59:35.466115 13042 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.40.61" port: 38411 } attempt: 3
I20260928 14:59:35.467321 13046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:35.466819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.61:38411 (local address 127.0.0.1:37160)
0928 14:59:35.467085 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:35.467174 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.61:38411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260928 14:59:35.468473 13046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:35.468100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.61:38411 (local address 127.0.0.1:37166)
0928 14:59:35.468246 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:35.468339 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.61:38411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 14:59:35.468710 13042 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.40.61:38411: Network error: Client connection negotiation failed: client connection to 127.31.40.61:38411: connect: Connection refused (error 111)
I20260928 14:59:35.612007 13042 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.40.61" port: 38411 } attempt: 4
I20260928 14:59:35.613355 13046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:35.612749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.61:38411 (local address 127.0.0.1:37174)
0928 14:59:35.613091 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:35.613203 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.61:38411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
I20260928 14:59:35.614609 13046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:35.614109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.61:38411 (local address 127.0.0.1:37178)
0928 14:59:35.614304 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:35.614433 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.61:38411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260928 14:59:35.614852 13042 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.40.61:38411: Network error: Client connection negotiation failed: client connection to 127.31.40.61:38411: connect: Connection refused (error 111)
W20260928 14:59:35.619688 13049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:59:35.620095 13049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:59:35.620254 13049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:59:35.627753 13049 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:59:35.627947 13049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:59:35.628057 13049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:59:35.628170 13049 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:59:35.628257 13049 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:59:35.638368 13049 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:33863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.40.62:36353,127.31.40.61:38411,127.31.40.60:38831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.61.keytab
--principal=kudu/127.31.40.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.61:38411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.40.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:59:35.639840 13049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:59:35.641216 13049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:59:35.653191 13054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:59:35.653379 13055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:59:35.653350 13057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:59:35.654495 13049 server_base.cc:1034] running on GCE node
I20260928 14:59:35.655361 13049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:59:35.657187 13049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:59:35.658447 13049 hybrid_clock.cc:648] HybridClock initialized: now 1790607575658374 us; error 44 us; skew 500 ppm
I20260928 14:59:35.658818 13049 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:59:35 dist-test-slave-j968 krb5kdc[12962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607575, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:59:35.662163 13049 init.cc:377] Logged in from keytab as kudu/127.31.40.61@KRBTEST.COM (short username kudu)
I20260928 14:59:35.663655 13049 webserver.cc:460] Webserver started at http://127.31.40.61:42715/ using document root <none> and password file <none>
I20260928 14:59:35.664424 13049 fs_manager.cc:359] Metadata directory not provided
I20260928 14:59:35.664567 13049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:59:35.664875 13049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:59:35.668098 13049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/minicluster-data/master-1/data/instance:
uuid: "e305e977c3324698bd373c935b455601"
format_stamp: "Formatted at 2026-09-28 14:59:35 on dist-test-slave-j968"
server_key: "365e9759eedb37b91985346ae1897724"
server_key_iv: "9d1f498bc9c3c562524de82b9bd828e8"
server_key_version: "encryptionkey@0"
I20260928 14:59:35.668941 13049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/minicluster-data/master-1/wal/instance:
uuid: "e305e977c3324698bd373c935b455601"
format_stamp: "Formatted at 2026-09-28 14:59:35 on dist-test-slave-j968"
server_key: "365e9759eedb37b91985346ae1897724"
server_key_iv: "9d1f498bc9c3c562524de82b9bd828e8"
server_key_version: "encryptionkey@0"
I20260928 14:59:35.674378 13049 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 14:59:35.678158 13064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:59:35.679549 13049 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:59:35.679730 13049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/minicluster-data/master-1/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/minicluster-data/master-1/wal
uuid: "e305e977c3324698bd373c935b455601"
format_stamp: "Formatted at 2026-09-28 14:59:35 on dist-test-slave-j968"
server_key: "365e9759eedb37b91985346ae1897724"
server_key_iv: "9d1f498bc9c3c562524de82b9bd828e8"
server_key_version: "encryptionkey@0"
I20260928 14:59:35.680002 13049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:59:35.697118 13049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:59:35.702364 13049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:59:35.702683 13049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:59:35.722375 13049 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.61:38411
I20260928 14:59:35.722397 13115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.61:38411 every 8 connection(s)
I20260928 14:59:35.723963 13049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/minicluster-data/master-1/data/info.pb
I20260928 14:59:35.726719 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 13049
I20260928 14:59:35.726929 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/minicluster-data/master-1/wal/instance
I20260928 14:59:35.727420 31904 external_mini_cluster.cc:1428] Setting key 1c74bd73c4f11d9333af1e40cba35d0e
I20260928 14:59:35.727895 13116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:59:35.735095 13116 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.40.62" port: 36353 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.31.40.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.60@KRBTEST.COM" created.
Sep 28 14:59:35 dist-test-slave-j968 krb5kdc[12962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607575, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.61@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:59:35.760207 13045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:35.738730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49986 (local address 127.31.40.62:36353)
0928 14:59:35.738959 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:35.738966 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:35.740461 (+ 1495us) server_negotiation.cc:407] Connection header received
0928 14:59:35.742479 (+ 2018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:35.742487 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:35.742598 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:35.742733 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:59:35.744744 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:35.745995 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:35.747785 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:35.748551 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:35.748707 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:35.753651 (+ 4944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:59:35.753657 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:59:35.753660 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:59:35.753699 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:59:35.756582 (+ 2883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:35.757647 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:35.757652 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:35.757654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:35.757788 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:35.758279 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:35.758282 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:35.758284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:35.758568 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:59:35.758730 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:35.759498 (+ 768us) server_negotiation.cc:299] Negotiation successful
0928 14:59:35.759772 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 14:59:35.761116 13119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:35.739326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:36353 (local address 127.0.0.1:49986)
0928 14:59:35.740244 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:35.740307 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 14:59:35.742154 (+ 1847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:59:35.743071 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:59:35.743081 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:59:35.743684 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:59:35.744502 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:35.744526 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:35.746204 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:35.746208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:35.747610 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:35.747625 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:35.748779 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:35.748785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:35.749007 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:35.749785 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:59:35.749809 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:59:35.753333 (+ 3524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:59:35.756755 (+ 3422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:59:35.756767 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:59:35.756789 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:59:35.757502 (+ 713us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:59:35.757905 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:59:35.757912 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:59:35.757919 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:59:35.758163 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:59:35.758711 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:59:35.758722 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:59:35.759099 (+ 377us) client_negotiation.cc:769] Sending connection context
0928 14:59:35.759330 (+ 231us) client_negotiation.cc:240] Negotiation successful
0928 14:59:35.759838 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":406,"threads_started":1}
I20260928 14:59:35.763101 13116 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.40.61" port: 38411 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:59:35 dist-test-slave-j968 krb5kdc[12962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607575, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.61@KRBTEST.COM for kudu/127.31.40.61@KRBTEST.COM
I20260928 14:59:35.778671 13119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:35.763991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.61:38411 (local address 127.0.0.1:37180)
0928 14:59:35.764179 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:35.764199 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 14:59:35.764310 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:59:35.765378 (+ 1068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:59:35.765382 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:59:35.765868 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:59:35.766017 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:35.766027 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:35.767376 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:35.767384 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:35.768306 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:35.768318 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:35.769156 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:35.769162 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:35.769357 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:35.769978 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:59:35.769997 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:59:35.772349 (+ 2352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:59:35.775884 (+ 3535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:59:35.775891 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:59:35.775895 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:59:35.776284 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:59:35.776712 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:59:35.776716 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:59:35.776720 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:59:35.776879 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:59:35.777582 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:59:35.777587 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:59:35.777746 (+ 159us) client_negotiation.cc:769] Sending connection context
0928 14:59:35.777919 (+ 173us) client_negotiation.cc:240] Negotiation successful
0928 14:59:35.778269 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 14:59:35.778807 13121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:35.764297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37180 (local address 127.31.40.61:38411)
0928 14:59:35.764953 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:35.764962 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:35.764990 (+ 28us) server_negotiation.cc:407] Connection header received
0928 14:59:35.765034 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:35.765040 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:35.765233 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:35.765396 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:59:35.766172 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:35.767239 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:35.768450 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:35.769007 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:35.769151 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:35.772575 (+ 3424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:59:35.772588 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:59:35.772603 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:59:35.772636 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:59:35.775669 (+ 3033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:35.776447 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:35.776454 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:35.776456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:35.776575 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:35.776999 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:35.777010 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:35.777012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:35.777418 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:59:35.777671 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:35.778077 (+ 406us) server_negotiation.cc:299] Negotiation successful
0928 14:59:35.778418 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":90,"server-negotiator.queue_time_us":505,"thread_start_us":363,"threads_started":1}
I20260928 14:59:35.781878 13116 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.40.60" port: 38831 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.60.keytab.
Entry for principal kudu/127.31.40.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.60.keytab.
I20260928 14:59:35.783265 13119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:35.782725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:38831 (local address 127.0.0.1:55672)
0928 14:59:35.782925 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:35.783112 (+ 187us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.60:38831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260928 14:59:35.783614 13065 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.40.60:38831: connect: Connection refused (error 111)
I20260928 14:59:35.784547 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.60:38831
--webserver_interface=127.31.40.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.60.keytab
--principal=kudu/127.31.40.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:33863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.40.62:36353,127.31.40.61:38411,127.31.40.60:38831
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:59:35.785539 13119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:35.785015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:38831 (local address 127.0.0.1:55688)
0928 14:59:35.785252 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:35.785371 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.60:38831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260928 14:59:35.785776 13116 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.40.60:38831: Network error: Client connection negotiation failed: client connection to 127.31.40.60:38831: connect: Connection refused (error 111)
I20260928 14:59:35.835196 13116 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.40.60" port: 38831 } attempt: 1
I20260928 14:59:35.836585 13119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:35.836063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:38831 (local address 127.0.0.1:55692)
0928 14:59:35.836292 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:35.836429 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.60:38831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260928 14:59:35.837823 13119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:35.837339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:38831 (local address 127.0.0.1:55694)
0928 14:59:35.837641 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:35.837714 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.60:38831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172}
W20260928 14:59:35.838027 13116 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.40.60:38831: Network error: Client connection negotiation failed: client connection to 127.31.40.60:38831: connect: Connection refused (error 111)
I20260928 14:59:35.906277 13116 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.40.60" port: 38831 } attempt: 2
I20260928 14:59:35.907629 13119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:35.907129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:38831 (local address 127.0.0.1:55708)
0928 14:59:35.907367 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:35.907462 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.60:38831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260928 14:59:35.908874 13119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:35.908469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:38831 (local address 127.0.0.1:55712)
0928 14:59:35.908643 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:35.908730 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.60:38831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260928 14:59:35.909099 13116 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.40.60:38831: Network error: Client connection negotiation failed: client connection to 127.31.40.60:38831: connect: Connection refused (error 111)
I20260928 14:59:35.914120 13042 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.40.61" port: 38411 } attempt: 5
Sep 28 14:59:35 dist-test-slave-j968 krb5kdc[12962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607575, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.62@KRBTEST.COM for kudu/127.31.40.61@KRBTEST.COM
I20260928 14:59:35.929832 13046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:35.914880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.61:38411 (local address 127.0.0.1:37184)
0928 14:59:35.915107 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:35.915126 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 14:59:35.915238 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:59:35.915660 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:59:35.915664 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:59:35.916191 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:59:35.916454 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:35.916471 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:35.918119 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:35.918124 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:35.919080 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:35.919102 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:35.920108 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:35.920113 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:35.920254 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:35.920904 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:59:35.920929 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:59:35.923256 (+ 2327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:59:35.927348 (+ 4092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:59:35.927353 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:59:35.927357 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:59:35.927754 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:59:35.928217 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:59:35.928220 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:59:35.928222 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:59:35.928357 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:59:35.928995 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:59:35.928998 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:59:35.929134 (+ 136us) client_negotiation.cc:769] Sending connection context
0928 14:59:35.929250 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 14:59:35.929484 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 14:59:35.930039 13121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:35.914969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37184 (local address 127.31.40.61:38411)
0928 14:59:35.915193 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:35.915201 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:35.915227 (+ 26us) server_negotiation.cc:407] Connection header received
0928 14:59:35.915390 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:35.915396 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:35.915487 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:35.915594 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:59:35.916658 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:35.917925 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:35.919238 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:35.919935 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:35.920056 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:35.923521 (+ 3465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:59:35.923530 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:59:35.923537 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:59:35.923590 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:59:35.927158 (+ 3568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:35.927909 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:35.927915 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:35.927919 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:35.928078 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:35.928496 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:35.928501 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:35.928504 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:35.928872 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:59:35.929016 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:35.929388 (+ 372us) server_negotiation.cc:299] Negotiation successful
0928 14:59:35.929642 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 14:59:35.931099 13042 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.40.60" port: 38831 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:59:35.932214 13046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:35.931741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:38831 (local address 127.0.0.1:55728)
0928 14:59:35.931937 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:35.932034 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.60:38831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 14:59:35.933233 13046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:35.932878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:38831 (local address 127.0.0.1:55732)
0928 14:59:35.933076 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:35.933146 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.60:38831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260928 14:59:35.933471 13042 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.40.60:38831: Network error: Client connection negotiation failed: client connection to 127.31.40.60:38831: connect: Connection refused (error 111)
I20260928 14:59:35.984709 13042 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.40.60" port: 38831 } attempt: 1
I20260928 14:59:35.985965 13046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:35.985492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:38831 (local address 127.0.0.1:55736)
0928 14:59:35.985699 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:35.985792 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.60:38831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260928 14:59:35.987052 13046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:35.986732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:38831 (local address 127.0.0.1:55746)
0928 14:59:35.986871 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:35.986946 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.60:38831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 14:59:35.987355 13042 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.40.60:38831: Network error: Client connection negotiation failed: client connection to 127.31.40.60:38831: connect: Connection refused (error 111)
I20260928 14:59:36.000325 13116 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.40.60" port: 38831 } attempt: 3
I20260928 14:59:36.001463 13119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:36.000967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:38831 (local address 127.0.0.1:55754)
0928 14:59:36.001200 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:36.001302 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.60:38831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 14:59:36.002575 13119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:36.002156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:38831 (local address 127.0.0.1:55758)
0928 14:59:36.002391 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:36.002468 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.60:38831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
W20260928 14:59:36.002776 13116 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.40.60:38831: Network error: Client connection negotiation failed: client connection to 127.31.40.60:38831: connect: Connection refused (error 111)
I20260928 14:59:36.055629 13042 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.40.60" port: 38831 } attempt: 2
I20260928 14:59:36.056951 13046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:36.056428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:38831 (local address 127.0.0.1:55770)
0928 14:59:36.056694 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:36.056785 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.60:38831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260928 14:59:36.057958 13046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:36.057618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:38831 (local address 127.0.0.1:55782)
0928 14:59:36.057762 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:36.057839 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.60:38831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 14:59:36.058161 13042 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.40.60:38831: Network error: Client connection negotiation failed: client connection to 127.31.40.60:38831: connect: Connection refused (error 111)
W20260928 14:59:36.123422 13123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:59:36.123769 13123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:59:36.123984 13123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:59:36.131249 13123 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:59:36.131408 13123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:59:36.131513 13123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:59:36.131611 13123 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:59:36.131695 13123 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:59:36.145469 13123 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:33863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.40.62:36353,127.31.40.61:38411,127.31.40.60:38831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.60.keytab
--principal=kudu/127.31.40.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.60:38831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.40.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:59:36.145990 13116 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.40.60" port: 38831 } attempt: 4
I20260928 14:59:36.147081 13123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:59:36.147126 13119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:36.146619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:38831 (local address 127.0.0.1:55790)
0928 14:59:36.146879 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:36.146970 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.60:38831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 14:59:36.148277 13119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:36.147855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:38831 (local address 127.0.0.1:55796)
0928 14:59:36.148063 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:36.148144 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.60:38831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260928 14:59:36.148517 13116 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.40.60:38831: Network error: Client connection negotiation failed: client connection to 127.31.40.60:38831: connect: Connection refused (error 111)
I20260928 14:59:36.148874 13123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:59:36.161621 13131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:59:36.161602 13129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:59:36.161733 13128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:59:36.162714 13123 server_base.cc:1034] running on GCE node
I20260928 14:59:36.163494 13123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:59:36.164350 13042 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.40.60" port: 38831 } attempt: 3
I20260928 14:59:36.165088 13123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:59:36.165484 13046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:36.164911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:38831 (local address 127.0.0.1:55806)
0928 14:59:36.165204 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:36.165306 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.60:38831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260928 14:59:36.166373 13123 hybrid_clock.cc:648] HybridClock initialized: now 1790607576166300 us; error 56 us; skew 500 ppm
I20260928 14:59:36.166567 13046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:36.166129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:38831 (local address 127.0.0.1:55812)
0928 14:59:36.166337 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:36.166423 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.60:38831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260928 14:59:36.166798 13042 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.40.60:38831: Network error: Client connection negotiation failed: client connection to 127.31.40.60:38831: connect: Connection refused (error 111)
I20260928 14:59:36.166819 13123 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:59:36 dist-test-slave-j968 krb5kdc[12962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607576, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:59:36.170068 13123 init.cc:377] Logged in from keytab as kudu/127.31.40.60@KRBTEST.COM (short username kudu)
I20260928 14:59:36.171612 13123 webserver.cc:460] Webserver started at http://127.31.40.60:46715/ using document root <none> and password file <none>
I20260928 14:59:36.172369 13123 fs_manager.cc:359] Metadata directory not provided
I20260928 14:59:36.172477 13123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:59:36.172730 13123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:59:36.175719 13123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/minicluster-data/master-2/data/instance:
uuid: "acd14e5b1be64dc9b3956fd4c34645d8"
format_stamp: "Formatted at 2026-09-28 14:59:36 on dist-test-slave-j968"
server_key: "4e50156b62b63c8745cdc51b985cac41"
server_key_iv: "acba499e66c0a25379ce9abcf1ce6f8d"
server_key_version: "encryptionkey@0"
I20260928 14:59:36.176581 13123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/minicluster-data/master-2/wal/instance:
uuid: "acd14e5b1be64dc9b3956fd4c34645d8"
format_stamp: "Formatted at 2026-09-28 14:59:36 on dist-test-slave-j968"
server_key: "4e50156b62b63c8745cdc51b985cac41"
server_key_iv: "acba499e66c0a25379ce9abcf1ce6f8d"
server_key_version: "encryptionkey@0"
I20260928 14:59:36.181836 13123 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 14:59:36.185642 13138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:59:36.187518 13123 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 14:59:36.187701 13123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/minicluster-data/master-2/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/minicluster-data/master-2/wal
uuid: "acd14e5b1be64dc9b3956fd4c34645d8"
format_stamp: "Formatted at 2026-09-28 14:59:36 on dist-test-slave-j968"
server_key: "4e50156b62b63c8745cdc51b985cac41"
server_key_iv: "acba499e66c0a25379ce9abcf1ce6f8d"
server_key_version: "encryptionkey@0"
I20260928 14:59:36.187985 13123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:59:36.200254 13123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:59:36.204641 13123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:59:36.204953 13123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:59:36.224172 13123 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.60:38831
I20260928 14:59:36.224195 13189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.60:38831 every 8 connection(s)
I20260928 14:59:36.225673 13123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/minicluster-data/master-2/data/info.pb
I20260928 14:59:36.227913 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 13123
I20260928 14:59:36.228120 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607365362981-31904-0/minicluster-data/master-2/wal/instance
I20260928 14:59:36.228708 31904 external_mini_cluster.cc:1428] Setting key 647a3f41489c16ad6fe7ef31b276866b
I20260928 14:59:36.230355 13190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:59:36.239570 13190 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.40.62" port: 36353 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:59:36 dist-test-slave-j968 krb5kdc[12962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.60@KRBTEST.COM
Sep 28 14:59:36 dist-test-slave-j968 krb5kdc[12962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607576, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.60@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:59:36.253109 13193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:36.230736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55822 (local address 127.31.40.60:38831)
0928 14:59:36.231561 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:36.231573 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:36.231615 (+ 42us) server_negotiation.cc:407] Connection header received
0928 14:59:36.233233 (+ 1618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:36.233257 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:36.233679 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:36.234031 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:59:36.235961 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:36.237296 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:36.239206 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:36.239928 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:36.240081 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:36.245067 (+ 4986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:59:36.245075 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:59:36.245086 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:59:36.245124 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:59:36.248252 (+ 3128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:36.248945 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:36.248951 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:36.248958 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:36.249258 (+ 300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:36.249675 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:36.249680 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:36.249683 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:36.250402 (+ 719us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:59:36.250648 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:36.251240 (+ 592us) server_negotiation.cc:299] Negotiation successful
0928 14:59:36.251715 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"spinlock_wait_cycles":20864,"thread_start_us":388,"threads_started":1}
I20260928 14:59:36.258989 13194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:36.242756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:36353 (local address 127.0.0.1:50002)
0928 14:59:36.243919 (+ 1163us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:36.243969 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 14:59:36.244119 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:59:36.244541 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:59:36.244546 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:59:36.245147 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:59:36.245466 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:36.245475 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:36.246521 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:36.246525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:36.247621 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:36.247633 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:36.248453 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:36.248460 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:36.248621 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:36.249190 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:59:36.249211 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:59:36.252213 (+ 3002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:59:36.256100 (+ 3887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:59:36.256111 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:59:36.256118 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:59:36.256631 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:59:36.257178 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:59:36.257183 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:59:36.257188 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:59:36.257373 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:59:36.258148 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:59:36.258154 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:59:36.258301 (+ 147us) client_negotiation.cc:769] Sending connection context
0928 14:59:36.258460 (+ 159us) client_negotiation.cc:240] Negotiation successful
0928 14:59:36.258697 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":897,"thread_start_us":721,"threads_started":1}
I20260928 14:59:36.259380 13045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:36.242969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50002 (local address 127.31.40.62:36353)
0928 14:59:36.243229 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:36.243238 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:36.244078 (+ 840us) server_negotiation.cc:407] Connection header received
0928 14:59:36.244246 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:36.244254 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:36.244353 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:36.244479 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:59:36.245631 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:36.246340 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:36.247763 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:36.248267 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:36.248412 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:36.252433 (+ 4021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:59:36.252441 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:59:36.252446 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:59:36.252491 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:59:36.255926 (+ 3435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:36.256788 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:36.256794 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:36.256798 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:36.256990 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:36.257523 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:36.257527 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:36.257530 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:36.257987 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:59:36.258190 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:36.258619 (+ 429us) server_negotiation.cc:299] Negotiation successful
0928 14:59:36.258891 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":93}
I20260928 14:59:36.260996 13190 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.40.61" port: 38411 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:59:36 dist-test-slave-j968 krb5kdc[12962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
Sep 28 14:59:36 dist-test-slave-j968 krb5kdc[12962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607576, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.60@KRBTEST.COM for kudu/127.31.40.61@KRBTEST.COM
I20260928 14:59:36.274622 13195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:36.256848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50010 (local address 127.31.40.62:36353)
0928 14:59:36.257501 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:36.257507 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:36.257528 (+ 21us) server_negotiation.cc:407] Connection header received
0928 14:59:36.257581 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:36.257584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:36.257638 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:36.257717 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:59:36.259665 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:36.260834 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:36.262445 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:36.263093 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:36.263233 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:36.268455 (+ 5222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:59:36.268460 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:59:36.268463 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:59:36.268495 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:59:36.271712 (+ 3217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:36.272443 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:36.272449 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:36.272453 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:36.272593 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:36.272963 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:36.272968 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:36.272972 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:36.273364 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:59:36.273574 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:36.273921 (+ 347us) server_negotiation.cc:299] Negotiation successful
0928 14:59:36.274160 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":477,"thread_start_us":270,"threads_started":1}
I20260928 14:59:36.276185 13194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:36.261854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.61:38411 (local address 127.0.0.1:37192)
0928 14:59:36.262024 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:36.262046 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 14:59:36.262139 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:59:36.262498 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:59:36.262505 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:59:36.262970 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:59:36.263164 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:36.263179 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:36.264547 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:36.264551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:36.265484 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:36.265496 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:36.266508 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:36.266516 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:36.266734 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:36.267299 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:59:36.267325 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:59:36.269637 (+ 2312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:59:36.273414 (+ 3777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:59:36.273443 (+ 29us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:59:36.273449 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:59:36.273973 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:59:36.274402 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:59:36.274407 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:59:36.274411 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:59:36.274608 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:59:36.275199 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:59:36.275204 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:59:36.275358 (+ 154us) client_negotiation.cc:769] Sending connection context
0928 14:59:36.275506 (+ 148us) client_negotiation.cc:240] Negotiation successful
0928 14:59:36.275801 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 14:59:36.276248 13121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:36.261877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37192 (local address 127.31.40.61:38411)
0928 14:59:36.262067 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:36.262073 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:36.262097 (+ 24us) server_negotiation.cc:407] Connection header received
0928 14:59:36.262247 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:36.262252 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:36.262341 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:36.262451 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:59:36.263308 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:36.264376 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:36.265638 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:36.266333 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:36.266469 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:36.269839 (+ 3370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:59:36.269848 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:59:36.269856 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:59:36.269896 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:59:36.273266 (+ 3370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:36.274093 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:36.274099 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:36.274103 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:36.274259 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:36.274754 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:36.274762 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:36.274767 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:36.275077 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:59:36.275245 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:36.275611 (+ 366us) server_negotiation.cc:299] Negotiation successful
0928 14:59:36.275822 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 14:59:36.277690 13190 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.40.60" port: 38831 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:59:36 dist-test-slave-j968 krb5kdc[12962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607576, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.60@KRBTEST.COM for kudu/127.31.40.60@KRBTEST.COM
Sep 28 14:59:36 dist-test-slave-j968 krb5kdc[12962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.61@KRBTEST.COM
I20260928 14:59:36.292265 13194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:36.278584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:38831 (local address 127.0.0.1:55836)
0928 14:59:36.278734 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:36.278757 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 14:59:36.278874 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:59:36.279276 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:59:36.279280 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:59:36.279639 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:59:36.279779 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:36.279787 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:36.280841 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:36.280850 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:36.281690 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:36.281701 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:36.282417 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:36.282423 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:36.282602 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:36.283125 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:59:36.283141 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:59:36.285493 (+ 2352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:59:36.289184 (+ 3691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:59:36.289188 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:59:36.289191 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:59:36.289605 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:59:36.290171 (+ 566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:59:36.290174 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:59:36.290177 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:59:36.290304 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:59:36.290999 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:59:36.291005 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:59:36.291180 (+ 175us) client_negotiation.cc:769] Sending connection context
0928 14:59:36.291302 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 14:59:36.291628 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 14:59:36.292547 13193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:36.278493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55836 (local address 127.31.40.60:38831)
0928 14:59:36.278668 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:36.278674 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:36.278821 (+ 147us) server_negotiation.cc:407] Connection header received
0928 14:59:36.279006 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:36.279011 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:36.279103 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:36.279204 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:59:36.279947 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:36.280716 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:36.281842 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:36.282262 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:36.282399 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:36.285721 (+ 3322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:59:36.285727 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:59:36.285732 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:59:36.285768 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:59:36.289024 (+ 3256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:36.289848 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:36.289862 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:36.289865 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:36.290052 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:36.290470 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:36.290476 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:36.290480 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:36.290886 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:59:36.291003 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:36.291462 (+ 459us) server_negotiation.cc:299] Negotiation successful
0928 14:59:36.292039 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 14:59:36.293756 13121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:36.276609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37198 (local address 127.31.40.61:38411)
0928 14:59:36.276774 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:36.276779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:36.276799 (+ 20us) server_negotiation.cc:407] Connection header received
0928 14:59:36.276848 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:36.276852 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:36.276917 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:36.277025 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:59:36.279109 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:36.280012 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:36.281294 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:36.281714 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:36.281859 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:36.286541 (+ 4682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:59:36.286550 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:59:36.286554 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:59:36.286594 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:59:36.289723 (+ 3129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:36.290786 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:36.290790 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:36.290793 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:36.291000 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:36.291477 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:36.291482 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:36.291485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:36.291851 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:59:36.292174 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:36.293246 (+ 1072us) server_negotiation.cc:299] Negotiation successful
0928 14:59:36.293444 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 14:59:36.298393 13190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acd14e5b1be64dc9b3956fd4c34645d8: Bootstrap starting.
I20260928 14:59:36.299120 31904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:59:36.302028 13190 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P acd14e5b1be64dc9b3956fd4c34645d8: Neither blocks nor log segments found. Creating new log.
I20260928 14:59:36.303104 13190 log.cc:824] T 00000000000000000000000000000000 P acd14e5b1be64dc9b3956fd4c34645d8: Log is configured to *not* fsync() on all Append() calls
I20260928 14:59:36.307299 13190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acd14e5b1be64dc9b3956fd4c34645d8: No bootstrap required, opened a new log
I20260928 14:59:36.323232 13190 raft_consensus.cc:348] T 00000000000000000000000000000000 P acd14e5b1be64dc9b3956fd4c34645d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e23f386b24452bb15a4efcdda75fea" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 36353 } } peers { permanent_uuid: "e305e977c3324698bd373c935b455601" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 38411 } } peers { permanent_uuid: "acd14e5b1be64dc9b3956fd4c34645d8" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 38831 } }
I20260928 14:59:36.323976 13190 raft_consensus.cc:374] T 00000000000000000000000000000000 P acd14e5b1be64dc9b3956fd4c34645d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:59:36.324174 13190 raft_consensus.cc:729] T 00000000000000000000000000000000 P acd14e5b1be64dc9b3956fd4c34645d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acd14e5b1be64dc9b3956fd4c34645d8, State: Initialized, Role: FOLLOWER
I20260928 14:59:36.325125 13190 consensus_queue.cc:260] T 00000000000000000000000000000000 P acd14e5b1be64dc9b3956fd4c34645d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e23f386b24452bb15a4efcdda75fea" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 36353 } } peers { permanent_uuid: "e305e977c3324698bd373c935b455601" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 38411 } } peers { permanent_uuid: "acd14e5b1be64dc9b3956fd4c34645d8" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 38831 } }
I20260928 14:59:36.327993 13121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:36.304670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37214 (local address 127.31.40.61:38411)
0928 14:59:36.304856 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:36.304862 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:36.305606 (+ 744us) server_negotiation.cc:407] Connection header received
0928 14:59:36.305971 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:36.305980 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:36.306086 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:36.306179 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:59:36.309390 (+ 3211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:36.310908 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:36.313957 (+ 3049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:36.314864 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:36.315004 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:36.320505 (+ 5501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:59:36.320512 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:59:36.320517 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:59:36.320567 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:59:36.324283 (+ 3716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:36.325280 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:36.325289 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:36.325293 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:36.325479 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:36.326054 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:36.326063 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:36.326069 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:36.326519 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:59:36.326752 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:36.327361 (+ 609us) server_negotiation.cc:299] Negotiation successful
0928 14:59:36.327637 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 14:59:36.328430 13190 sys_catalog.cc:564] T 00000000000000000000000000000000 P acd14e5b1be64dc9b3956fd4c34645d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:59:36.329751 13204 sys_catalog.cc:455] T 00000000000000000000000000000000 P acd14e5b1be64dc9b3956fd4c34645d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e23f386b24452bb15a4efcdda75fea" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 36353 } } peers { permanent_uuid: "e305e977c3324698bd373c935b455601" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 38411 } } peers { permanent_uuid: "acd14e5b1be64dc9b3956fd4c34645d8" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 38831 } } }
I20260928 14:59:36.330169 13204 sys_catalog.cc:458] T 00000000000000000000000000000000 P acd14e5b1be64dc9b3956fd4c34645d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:59:36.339468 13195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:36.304132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50020 (local address 127.31.40.62:36353)
0928 14:59:36.304370 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:36.304378 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:36.320474 (+ 16096us) server_negotiation.cc:407] Connection header received
0928 14:59:36.320727 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:36.320733 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:36.320819 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:36.320909 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:59:36.323068 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:36.324236 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:36.326968 (+ 2732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:36.327634 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:36.327762 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:36.331780 (+ 4018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:59:36.331789 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:59:36.331794 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:59:36.331839 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:59:36.335680 (+ 3841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:36.336907 (+ 1227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:36.336913 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:36.336917 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:36.337051 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:36.337462 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:36.337467 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:36.337471 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:36.337875 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:59:36.338174 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:36.338902 (+ 728us) server_negotiation.cc:299] Negotiation successful
0928 14:59:36.339121 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 14:59:36.339917 13193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:36.304655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55850 (local address 127.31.40.60:38831)
0928 14:59:36.304889 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:36.304904 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:36.320029 (+ 15125us) server_negotiation.cc:407] Connection header received
0928 14:59:36.320591 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:36.320596 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:36.320714 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:36.320810 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:59:36.323093 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:36.324236 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:36.326536 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:36.327244 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:36.327366 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:36.331673 (+ 4307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:59:36.331681 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:59:36.331689 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:59:36.331733 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:59:36.335629 (+ 3896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:36.336909 (+ 1280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:36.336914 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:36.336917 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:36.337091 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:36.337612 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:36.337617 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:36.337620 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:36.338048 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:59:36.338338 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:36.339292 (+ 954us) server_negotiation.cc:299] Negotiation successful
0928 14:59:36.339537 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
W20260928 14:59:36.345930 13215 catalog_manager.cc:1558] T 00000000000000000000000000000000 P acd14e5b1be64dc9b3956fd4c34645d8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:59:36.346076 13215 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:59:36.347975 13042 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.40.60" port: 38831 } attempt: 4
I20260928 14:59:36.343616 13123 master_runner.cc:186] Error getting master registration for 127.31.40.62:36353: OK, instance_id { permanent_uuid: "b8e23f386b24452bb15a4efcdda75fea" instance_seqno: 1790607575193362 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 14:59:36.352975 13123 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 28 14:59:36 dist-test-slave-j968 krb5kdc[12962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607575, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.62@KRBTEST.COM for kudu/127.31.40.60@KRBTEST.COM
I20260928 14:59:36.364601 13193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:36.348755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55866 (local address 127.31.40.60:38831)
0928 14:59:36.348934 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:36.348952 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:36.348970 (+ 18us) server_negotiation.cc:407] Connection header received
0928 14:59:36.349019 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:36.349023 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:36.349093 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:36.349207 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:59:36.350162 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:36.351511 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:36.353330 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:36.354016 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:36.354148 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:36.358119 (+ 3971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:59:36.358126 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:59:36.358131 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:59:36.358170 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:59:36.361680 (+ 3510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:36.362541 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:36.362544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:36.362547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:36.362681 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:36.363122 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:36.363126 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:36.363129 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:36.363455 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:59:36.363629 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:36.364115 (+ 486us) server_negotiation.cc:299] Negotiation successful
0928 14:59:36.364312 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 14:59:36.364740 13046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:36.348548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:38831 (local address 127.0.0.1:55866)
0928 14:59:36.348745 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:36.348765 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 14:59:36.348878 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:59:36.349252 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:59:36.349257 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:59:36.349759 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:59:36.350002 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:36.350018 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:36.351674 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:36.351680 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:36.353196 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:36.353211 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:36.354178 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:36.354183 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:36.354304 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:36.354972 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:59:36.355000 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:59:36.357899 (+ 2899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:59:36.361857 (+ 3958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:59:36.361863 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:59:36.361868 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:59:36.362409 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:59:36.362792 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:59:36.362797 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:59:36.362801 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:59:36.362974 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:59:36.363617 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:59:36.363623 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:59:36.363821 (+ 198us) client_negotiation.cc:769] Sending connection context
0928 14:59:36.364012 (+ 191us) client_negotiation.cc:240] Negotiation successful
0928 14:59:36.364352 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 14:59:36.370853 13042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8e23f386b24452bb15a4efcdda75fea: Bootstrap starting.
I20260928 14:59:36.375181 13042 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8e23f386b24452bb15a4efcdda75fea: Neither blocks nor log segments found. Creating new log.
I20260928 14:59:36.376605 13042 log.cc:824] T 00000000000000000000000000000000 P b8e23f386b24452bb15a4efcdda75fea: Log is configured to *not* fsync() on all Append() calls
I20260928 14:59:36.380053 13042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8e23f386b24452bb15a4efcdda75fea: No bootstrap required, opened a new log
I20260928 14:59:36.391554 13042 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8e23f386b24452bb15a4efcdda75fea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e23f386b24452bb15a4efcdda75fea" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 36353 } } peers { permanent_uuid: "e305e977c3324698bd373c935b455601" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 38411 } } peers { permanent_uuid: "acd14e5b1be64dc9b3956fd4c34645d8" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 38831 } }
I20260928 14:59:36.392129 13042 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8e23f386b24452bb15a4efcdda75fea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:59:36.392251 13042 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8e23f386b24452bb15a4efcdda75fea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8e23f386b24452bb15a4efcdda75fea, State: Initialized, Role: FOLLOWER
I20260928 14:59:36.393108 13042 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8e23f386b24452bb15a4efcdda75fea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e23f386b24452bb15a4efcdda75fea" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 36353 } } peers { permanent_uuid: "e305e977c3324698bd373c935b455601" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 38411 } } peers { permanent_uuid: "acd14e5b1be64dc9b3956fd4c34645d8" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 38831 } }
I20260928 14:59:36.394132 13204 raft_consensus.cc:482] T 00000000000000000000000000000000 P acd14e5b1be64dc9b3956fd4c34645d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:59:36.394464 13204 raft_consensus.cc:504] T 00000000000000000000000000000000 P acd14e5b1be64dc9b3956fd4c34645d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e23f386b24452bb15a4efcdda75fea" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 36353 } } peers { permanent_uuid: "e305e977c3324698bd373c935b455601" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 38411 } } peers { permanent_uuid: "acd14e5b1be64dc9b3956fd4c34645d8" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 38831 } }
I20260928 14:59:36.395004 13218 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8e23f386b24452bb15a4efcdda75fea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e23f386b24452bb15a4efcdda75fea" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 36353 } } peers { permanent_uuid: "e305e977c3324698bd373c935b455601" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 38411 } } peers { permanent_uuid: "acd14e5b1be64dc9b3956fd4c34645d8" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 38831 } } }
I20260928 14:59:36.395522 13218 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8e23f386b24452bb15a4efcdda75fea [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:59:36.395594 13042 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8e23f386b24452bb15a4efcdda75fea [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:59:36.396315 13204 leader_election.cc:288] T 00000000000000000000000000000000 P acd14e5b1be64dc9b3956fd4c34645d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8e23f386b24452bb15a4efcdda75fea (127.31.40.62:36353), e305e977c3324698bd373c935b455601 (127.31.40.61:38411)
I20260928 14:59:36.397064 13017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "acd14e5b1be64dc9b3956fd4c34645d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8e23f386b24452bb15a4efcdda75fea" is_pre_election: true
I20260928 14:59:36.397723 13017 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b8e23f386b24452bb15a4efcdda75fea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acd14e5b1be64dc9b3956fd4c34645d8 in term 0.
I20260928 14:59:36.398430 13140 leader_election.cc:302] T 00000000000000000000000000000000 P acd14e5b1be64dc9b3956fd4c34645d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acd14e5b1be64dc9b3956fd4c34645d8, b8e23f386b24452bb15a4efcdda75fea; no voters:
I20260928 14:59:36.398929 13204 raft_consensus.cc:2781] T 00000000000000000000000000000000 P acd14e5b1be64dc9b3956fd4c34645d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:59:36.399112 13204 raft_consensus.cc:482] T 00000000000000000000000000000000 P acd14e5b1be64dc9b3956fd4c34645d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:59:36.399268 13204 raft_consensus.cc:3037] T 00000000000000000000000000000000 P acd14e5b1be64dc9b3956fd4c34645d8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:59:36.404326 13204 raft_consensus.cc:504] T 00000000000000000000000000000000 P acd14e5b1be64dc9b3956fd4c34645d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e23f386b24452bb15a4efcdda75fea" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 36353 } } peers { permanent_uuid: "e305e977c3324698bd373c935b455601" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 38411 } } peers { permanent_uuid: "acd14e5b1be64dc9b3956fd4c34645d8" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 38831 } }
I20260928 14:59:36.407145 13204 leader_election.cc:288] T 00000000000000000000000000000000 P acd14e5b1be64dc9b3956fd4c34645d8 [CANDIDATE]: Term 1 election: Requested vote from peers b8e23f386b24452bb15a4efcdda75fea (127.31.40.62:36353), e305e977c3324698bd373c935b455601 (127.31.40.61:38411)
I20260928 14:59:36.407482 13017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "acd14e5b1be64dc9b3956fd4c34645d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8e23f386b24452bb15a4efcdda75fea"
I20260928 14:59:36.407910 13017 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8e23f386b24452bb15a4efcdda75fea [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:59:36.412147 13017 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b8e23f386b24452bb15a4efcdda75fea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acd14e5b1be64dc9b3956fd4c34645d8 in term 1.
I20260928 14:59:36.412700 13140 leader_election.cc:302] T 00000000000000000000000000000000 P acd14e5b1be64dc9b3956fd4c34645d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acd14e5b1be64dc9b3956fd4c34645d8, b8e23f386b24452bb15a4efcdda75fea; no voters:
I20260928 14:59:36.413112 13204 raft_consensus.cc:2781] T 00000000000000000000000000000000 P acd14e5b1be64dc9b3956fd4c34645d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:59:36.413924 13204 raft_consensus.cc:686] T 00000000000000000000000000000000 P acd14e5b1be64dc9b3956fd4c34645d8 [term 1 LEADER]: Becoming Leader. State: Replica: acd14e5b1be64dc9b3956fd4c34645d8, State: Running, Role: LEADER
I20260928 14:59:36.414637 13204 consensus_queue.cc:237] T 00000000000000000000000000000000 P acd14e5b1be64dc9b3956fd4c34645d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e23f386b24452bb15a4efcdda75fea" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 36353 } } peers { permanent_uuid: "e305e977c3324698bd373c935b455601" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 38411 } } peers { permanent_uuid: "acd14e5b1be64dc9b3956fd4c34645d8" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 38831 } }
I20260928 14:59:36.426883 13226 sys_catalog.cc:455] T 00000000000000000000000000000000 P acd14e5b1be64dc9b3956fd4c34645d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader acd14e5b1be64dc9b3956fd4c34645d8. Latest consensus state: current_term: 1 leader_uuid: "acd14e5b1be64dc9b3956fd4c34645d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e23f386b24452bb15a4efcdda75fea" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 36353 } } peers { permanent_uuid: "e305e977c3324698bd373c935b455601" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 38411 } } peers { permanent_uuid: "acd14e5b1be64dc9b3956fd4c34645d8" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 38831 } } }
I20260928 14:59:36.427404 13226 sys_catalog.cc:458] T 00000000000000000000000000000000 P acd14e5b1be64dc9b3956fd4c34645d8 [sys.catalog]: This master's current role is: LEADER
I20260928 14:59:36.428938 13231 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260928 14:59:36.430253 12996 tablet.cc:2369] T 00000000000000000000000000000000 P b8e23f386b24452bb15a4efcdda75fea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:59:36.432314 13231 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:59:36.435252 12975 master_runner.cc:186] Error getting master registration for 127.31.40.61:38411: OK, instance_id { permanent_uuid: "e305e977c3324698bd373c935b455601" instance_seqno: 1790607575716595 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 14:59:36.446762 12975 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 14:59:36.447767 13116 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.40.60" port: 38831 } attempt: 5
W20260928 14:59:36.450579 13232 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b8e23f386b24452bb15a4efcdda75fea: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:59:36.450875 13232 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260928 14:59:36.457201 13144 tablet.cc:2369] T 00000000000000000000000000000000 P acd14e5b1be64dc9b3956fd4c34645d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 14:59:36 dist-test-slave-j968 krb5kdc[12962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607575, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.61@KRBTEST.COM for kudu/127.31.40.60@KRBTEST.COM
I20260928 14:59:36.462514 13017 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b8e23f386b24452bb15a4efcdda75fea [term 1 FOLLOWER]: Refusing update from remote peer acd14e5b1be64dc9b3956fd4c34645d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:59:36.463495 13226 consensus_queue.cc:1035] T 00000000000000000000000000000000 P acd14e5b1be64dc9b3956fd4c34645d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8e23f386b24452bb15a4efcdda75fea" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 36353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:59:36.479552 13119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:36.448434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:38831 (local address 127.0.0.1:55872)
0928 14:59:36.448744 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:36.448764 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 14:59:36.448881 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:59:36.449551 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:59:36.449557 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:59:36.450105 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:59:36.450317 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:36.450329 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:36.451917 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:36.451925 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:36.453555 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:36.453575 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:36.454671 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:36.454706 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:36.455068 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:36.456190 (+ 1122us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:59:36.456245 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:59:36.460791 (+ 4546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:59:36.471569 (+ 10778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:59:36.471577 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:59:36.471581 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:59:36.472866 (+ 1285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:59:36.473705 (+ 839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:59:36.473711 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:59:36.473773 (+ 62us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:59:36.474101 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:59:36.475411 (+ 1310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:59:36.475454 (+ 43us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:59:36.475785 (+ 331us) client_negotiation.cc:769] Sending connection context
0928 14:59:36.476358 (+ 573us) client_negotiation.cc:240] Negotiation successful
0928 14:59:36.477635 (+ 1277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172}
I20260928 14:59:36.480052 13193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:36.448934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55872 (local address 127.31.40.60:38831)
0928 14:59:36.449141 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:36.449148 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:36.449171 (+ 23us) server_negotiation.cc:407] Connection header received
0928 14:59:36.449242 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:36.449248 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:36.449353 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:36.449508 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:59:36.450508 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:36.451672 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:36.453733 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:36.454475 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:36.454607 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:36.461097 (+ 6490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:59:36.461104 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:59:36.461109 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:59:36.461151 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:59:36.471176 (+ 10025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:36.473063 (+ 1887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:36.473068 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:36.473106 (+ 38us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:36.473407 (+ 301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:36.474301 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:36.474306 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:36.474311 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:36.475178 (+ 867us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:59:36.476422 (+ 1244us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:36.477272 (+ 850us) server_negotiation.cc:299] Negotiation successful
0928 14:59:36.478392 (+ 1120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 14:59:36.486193 13116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e305e977c3324698bd373c935b455601: Bootstrap starting.
I20260928 14:59:36.495368 13116 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e305e977c3324698bd373c935b455601: Neither blocks nor log segments found. Creating new log.
I20260928 14:59:36.498574 13116 log.cc:824] T 00000000000000000000000000000000 P e305e977c3324698bd373c935b455601: Log is configured to *not* fsync() on all Append() calls
I20260928 14:59:36.503221 13218 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8e23f386b24452bb15a4efcdda75fea [sys.catalog]: SysCatalogTable state changed. Reason: New leader acd14e5b1be64dc9b3956fd4c34645d8. Latest consensus state: current_term: 1 leader_uuid: "acd14e5b1be64dc9b3956fd4c34645d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e23f386b24452bb15a4efcdda75fea" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 36353 } } peers { permanent_uuid: "e305e977c3324698bd373c935b455601" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 38411 } } peers { permanent_uuid: "acd14e5b1be64dc9b3956fd4c34645d8" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 38831 } } }
I20260928 14:59:36.503587 13218 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8e23f386b24452bb15a4efcdda75fea [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:59:36.509264 13234 mvcc.cc:205] Tried to move back new op lower bound from 7334328633123942400 to 7334328633000407040. Current Snapshot: MvccSnapshot[applied={T|T < 7334328633123942400}]
I20260928 14:59:36.517963 13116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e305e977c3324698bd373c935b455601: No bootstrap required, opened a new log
I20260928 14:59:36.527218 13218 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8e23f386b24452bb15a4efcdda75fea [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "acd14e5b1be64dc9b3956fd4c34645d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e23f386b24452bb15a4efcdda75fea" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 36353 } } peers { permanent_uuid: "e305e977c3324698bd373c935b455601" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 38411 } } peers { permanent_uuid: "acd14e5b1be64dc9b3956fd4c34645d8" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 38831 } } }
I20260928 14:59:36.527527 13218 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8e23f386b24452bb15a4efcdda75fea [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:59:36.533654 13226 sys_catalog.cc:455] T 00000000000000000000000000000000 P acd14e5b1be64dc9b3956fd4c34645d8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "acd14e5b1be64dc9b3956fd4c34645d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e23f386b24452bb15a4efcdda75fea" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 36353 } } peers { permanent_uuid: "e305e977c3324698bd373c935b455601" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 38411 } } peers { permanent_uuid: "acd14e5b1be64dc9b3956fd4c34645d8" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 38831 } } }
I20260928 14:59:36.533974 13226 sys_catalog.cc:458] T 00000000000000000000000000000000 P acd14e5b1be64dc9b3956fd4c34645d8 [sys.catalog]: This master's current role is: LEADER
I20260928 14:59:36.540740 13116 raft_consensus.cc:348] T 00000000000000000000000000000000 P e305e977c3324698bd373c935b455601 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e23f386b24452bb15a4efcdda75fea" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 36353 } } peers { permanent_uuid: "e305e977c3324698bd373c935b455601" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 38411 } } peers { permanent_uuid: "acd14e5b1be64dc9b3956fd4c34645d8" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 38831 } }
I20260928 14:59:36.541226 13116 raft_consensus.cc:374] T 00000000000000000000000000000000 P e305e977c3324698bd373c935b455601 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:59:36.541354 13116 raft_consensus.cc:729] T 00000000000000000000000000000000 P e305e977c3324698bd373c935b455601 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e305e977c3324698bd373c935b455601, State: Initialized, Role: FOLLOWER
I20260928 14:59:36.542152 13116 consensus_queue.cc:260] T 00000000000000000000000000000000 P e305e977c3324698bd373c935b455601 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e23f386b24452bb15a4efcdda75fea" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 36353 } } peers { permanent_uuid: "e305e977c3324698bd373c935b455601" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 38411 } } peers { permanent_uuid: "acd14e5b1be64dc9b3956fd4c34645d8" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 38831 } }
I20260928 14:59:36.543960 13231 catalog_manager.cc:1347] Generated new cluster ID: 7212799020b34a47b10ce98c7b299e5f
I20260928 14:59:36.544078 13231 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:59:36.543633 13240 sys_catalog.cc:455] T 00000000000000000000000000000000 P e305e977c3324698bd373c935b455601 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e23f386b24452bb15a4efcdda75fea" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 36353 } } peers { permanent_uuid: "e305e977c3324698bd373c935b455601" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 38411 } } peers { permanent_uuid: "acd14e5b1be64dc9b3956fd4c34645d8" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 38831 } } }
I20260928 14:59:36.544672 13240 sys_catalog.cc:458] T 00000000000000000000000000000000 P e305e977c3324698bd373c935b455601 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:59:36.545935 13116 sys_catalog.cc:564] T 00000000000000000000000000000000 P e305e977c3324698bd373c935b455601 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:59:36.545989 13090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "acd14e5b1be64dc9b3956fd4c34645d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e305e977c3324698bd373c935b455601"
I20260928 14:59:36.546479 13090 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e305e977c3324698bd373c935b455601 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:59:36.550158 13090 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e305e977c3324698bd373c935b455601 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acd14e5b1be64dc9b3956fd4c34645d8 in term 1.
I20260928 14:59:36.552573 13091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "acd14e5b1be64dc9b3956fd4c34645d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e305e977c3324698bd373c935b455601" is_pre_election: true
I20260928 14:59:36.552929 13091 raft_consensus.cc:2362] T 00000000000000000000000000000000 P e305e977c3324698bd373c935b455601 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate acd14e5b1be64dc9b3956fd4c34645d8 in term 1. Re-sending same reply.
I20260928 14:59:36.553897 13089 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e305e977c3324698bd373c935b455601 [term 1 FOLLOWER]: Refusing update from remote peer acd14e5b1be64dc9b3956fd4c34645d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:59:36.556038 13204 consensus_queue.cc:1035] T 00000000000000000000000000000000 P acd14e5b1be64dc9b3956fd4c34645d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e305e977c3324698bd373c935b455601" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 38411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:59:36.601330 13240 sys_catalog.cc:455] T 00000000000000000000000000000000 P e305e977c3324698bd373c935b455601 [sys.catalog]: SysCatalogTable state changed. Reason: New leader acd14e5b1be64dc9b3956fd4c34645d8. Latest consensus state: current_term: 1 leader_uuid: "acd14e5b1be64dc9b3956fd4c34645d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e23f386b24452bb15a4efcdda75fea" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 36353 } } peers { permanent_uuid: "e305e977c3324698bd373c935b455601" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 38411 } } peers { permanent_uuid: "acd14e5b1be64dc9b3956fd4c34645d8" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 38831 } } }
I20260928 14:59:36.601706 13240 sys_catalog.cc:458] T 00000000000000000000000000000000 P e305e977c3324698bd373c935b455601 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:59:36.606874 13231 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:59:36.606899 13247 sys_catalog.cc:455] T 00000000000000000000000000000000 P e305e977c3324698bd373c935b455601 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "acd14e5b1be64dc9b3956fd4c34645d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e23f386b24452bb15a4efcdda75fea" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 36353 } } peers { permanent_uuid: "e305e977c3324698bd373c935b455601" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 38411 } } peers { permanent_uuid: "acd14e5b1be64dc9b3956fd4c34645d8" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 38831 } } }
I20260928 14:59:36.607419 13247 sys_catalog.cc:458] T 00000000000000000000000000000000 P e305e977c3324698bd373c935b455601 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:59:36.608604 13226 sys_catalog.cc:455] T 00000000000000000000000000000000 P acd14e5b1be64dc9b3956fd4c34645d8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "acd14e5b1be64dc9b3956fd4c34645d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e23f386b24452bb15a4efcdda75fea" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 36353 } } peers { permanent_uuid: "e305e977c3324698bd373c935b455601" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 38411 } } peers { permanent_uuid: "acd14e5b1be64dc9b3956fd4c34645d8" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 38831 } } }
I20260928 14:59:36.608922 13226 sys_catalog.cc:458] T 00000000000000000000000000000000 P acd14e5b1be64dc9b3956fd4c34645d8 [sys.catalog]: This master's current role is: LEADER
I20260928 14:59:36.608930 13231 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:59:36.640045 13231 catalog_manager.cc:5868] T 00000000000000000000000000000000 P acd14e5b1be64dc9b3956fd4c34645d8: Generated new TSK 0
I20260928 14:59:36.641531 13231 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:59:36.704727 13256 catalog_manager.cc:1259] Loaded cluster ID: 7212799020b34a47b10ce98c7b299e5f
I20260928 14:59:36.704867 13256 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e305e977c3324698bd373c935b455601: loading cluster ID for follower catalog manager: success
I20260928 14:59:36.708895 13256 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e305e977c3324698bd373c935b455601: acquiring CA information for follower catalog manager: success
I20260928 14:59:36.711684 13256 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e305e977c3324698bd373c935b455601: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 14:59:37.453013 13232 catalog_manager.cc:1259] Loaded cluster ID: 7212799020b34a47b10ce98c7b299e5f
I20260928 14:59:37.453114 13232 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b8e23f386b24452bb15a4efcdda75fea: loading cluster ID for follower catalog manager: success
I20260928 14:59:37.455683 13232 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b8e23f386b24452bb15a4efcdda75fea: acquiring CA information for follower catalog manager: success
I20260928 14:59:37.457609 13232 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b8e23f386b24452bb15a4efcdda75fea: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 14:59:37.807458 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 12975
I20260928 14:59:37.809247 13037 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:59:37.920815 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 12975
I20260928 14:59:37.940411 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 13049
I20260928 14:59:37.941579 13111 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:59:38.049952 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 13049
I20260928 14:59:38.071346 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 13123
I20260928 14:59:38.072607 13185 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:59:38.189194 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 13123
2026-09-28T14:59:38Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4828 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607365362981-31904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:59:38 dist-test-slave-j968 krb5kdc[13266](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:59:38 dist-test-slave-j968 krb5kdc[13266](info): set up 2 sockets
Sep 28 14:59:38 dist-test-slave-j968 krb5kdc[13266](info): commencing operation
krb5kdc: starting...
W20260928 14:59:39.309356 31904 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Sep 28 14:59:39 dist-test-slave-j968 krb5kdc[13266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:59:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:59:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab.
Entry for principal kudu/127.31.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab.
I20260928 14:59:39.565752 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607365362981-31904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607365362981-31904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607365362981-31904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:34493
--webserver_interface=127.31.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:38971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.40.62:34493
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:59:39.915115 13279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:59:39.915446 13279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:59:39.915624 13279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:59:39.923141 13279 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:59:39.923262 13279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:59:39.923372 13279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:59:39.923483 13279 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:59:39.923573 13279 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:59:39.934093 13279 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:38971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607365362981-31904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607365362981-31904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.40.62:34493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:34493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607365362981-31904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:59:39.935606 13279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:59:39.937192 13279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:59:39.950838 13287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:59:39.950910 13285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:59:39.950886 13284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:59:39.951907 13279 server_base.cc:1034] running on GCE node
I20260928 14:59:39.952829 13279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:59:39.954576 13279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:59:39.955951 13279 hybrid_clock.cc:648] HybridClock initialized: now 1790607579955807 us; error 86 us; skew 500 ppm
I20260928 14:59:39.956458 13279 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:59:39 dist-test-slave-j968 krb5kdc[13266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607579, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:59:39.960366 13279 init.cc:377] Logged in from keytab as kudu/127.31.40.62@KRBTEST.COM (short username kudu)
I20260928 14:59:39.962308 13279 webserver.cc:460] Webserver started at http://127.31.40.62:32865/ using document root <none> and password file <none>
I20260928 14:59:39.963145 13279 fs_manager.cc:359] Metadata directory not provided
I20260928 14:59:39.963310 13279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:59:39.963649 13279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:59:39.967026 13279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607365362981-31904-0/minicluster-data/master-0/data/instance:
uuid: "65a68ee1eb024af699fad9e16b47c5c0"
format_stamp: "Formatted at 2026-09-28 14:59:39 on dist-test-slave-j968"
server_key: "f31a7f5b7f55796c986a546d291908bc"
server_key_iv: "b5abfa7a9a5b5947922ed8589cea5c8b"
server_key_version: "encryptionkey@0"
I20260928 14:59:39.967973 13279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607365362981-31904-0/minicluster-data/master-0/wal/instance:
uuid: "65a68ee1eb024af699fad9e16b47c5c0"
format_stamp: "Formatted at 2026-09-28 14:59:39 on dist-test-slave-j968"
server_key: "f31a7f5b7f55796c986a546d291908bc"
server_key_iv: "b5abfa7a9a5b5947922ed8589cea5c8b"
server_key_version: "encryptionkey@0"
I20260928 14:59:39.973830 13279 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 14:59:39.977924 13294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:59:39.979564 13279 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 14:59:39.979806 13279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607365362981-31904-0/minicluster-data/master-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607365362981-31904-0/minicluster-data/master-0/wal
uuid: "65a68ee1eb024af699fad9e16b47c5c0"
format_stamp: "Formatted at 2026-09-28 14:59:39 on dist-test-slave-j968"
server_key: "f31a7f5b7f55796c986a546d291908bc"
server_key_iv: "b5abfa7a9a5b5947922ed8589cea5c8b"
server_key_version: "encryptionkey@0"
I20260928 14:59:39.980114 13279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607365362981-31904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607365362981-31904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607365362981-31904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:59:40.007090 13279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:59:40.011574 13279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:59:40.011902 13279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:59:40.031431 13279 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.62:34493
I20260928 14:59:40.031520 13345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.62:34493 every 8 connection(s)
I20260928 14:59:40.032912 13279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
I20260928 14:59:40.037137 13346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:59:40.041384 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 13279
I20260928 14:59:40.041648 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607365362981-31904-0/minicluster-data/master-0/wal/instance
I20260928 14:59:40.042505 31904 external_mini_cluster.cc:1428] Setting key d9305571557f5346b2407e4703332296
I20260928 14:59:40.051736 13346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65a68ee1eb024af699fad9e16b47c5c0: Bootstrap starting.
I20260928 14:59:40.055860 13346 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65a68ee1eb024af699fad9e16b47c5c0: Neither blocks nor log segments found. Creating new log.
I20260928 14:59:40.057389 13346 log.cc:824] T 00000000000000000000000000000000 P 65a68ee1eb024af699fad9e16b47c5c0: Log is configured to *not* fsync() on all Append() calls
Sep 28 14:59:40 dist-test-slave-j968 krb5kdc[13266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:59:40.060954 13346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65a68ee1eb024af699fad9e16b47c5c0: No bootstrap required, opened a new log
I20260928 14:59:40.067605 13349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:40.044625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35284 (local address 127.31.40.62:34493)
0928 14:59:40.045518 (+ 893us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:40.045531 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:40.045573 (+ 42us) server_negotiation.cc:407] Connection header received
0928 14:59:40.047217 (+ 1644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:40.047240 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:40.047685 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:40.048096 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:59:40.049978 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:40.051316 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:40.053285 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:40.054080 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:40.054215 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:40.059465 (+ 5250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:59:40.059474 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:59:40.059491 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:59:40.059534 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:59:40.063021 (+ 3487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:40.063936 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:40.063944 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:40.063951 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:40.064159 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:40.064601 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:40.064604 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:40.064606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:40.065173 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:59:40.065524 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:40.066025 (+ 501us) server_negotiation.cc:299] Negotiation successful
0928 14:59:40.066483 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":678,"thread_start_us":465,"threads_started":1}
I20260928 14:59:40.069391 13346 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65a68ee1eb024af699fad9e16b47c5c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a68ee1eb024af699fad9e16b47c5c0" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34493 } }
I20260928 14:59:40.069778 13346 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65a68ee1eb024af699fad9e16b47c5c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:59:40.069908 13346 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65a68ee1eb024af699fad9e16b47c5c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65a68ee1eb024af699fad9e16b47c5c0, State: Initialized, Role: FOLLOWER
I20260928 14:59:40.070626 13346 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65a68ee1eb024af699fad9e16b47c5c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a68ee1eb024af699fad9e16b47c5c0" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34493 } }
I20260928 14:59:40.070922 13346 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65a68ee1eb024af699fad9e16b47c5c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:59:40.071075 13346 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65a68ee1eb024af699fad9e16b47c5c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:59:40.071269 13346 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65a68ee1eb024af699fad9e16b47c5c0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:59:40.071743 31904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:59:40.074600 13346 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65a68ee1eb024af699fad9e16b47c5c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a68ee1eb024af699fad9e16b47c5c0" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34493 } }
I20260928 14:59:40.075263 13346 leader_election.cc:302] T 00000000000000000000000000000000 P 65a68ee1eb024af699fad9e16b47c5c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65a68ee1eb024af699fad9e16b47c5c0; no voters:
I20260928 14:59:40.076035 13346 leader_election.cc:288] T 00000000000000000000000000000000 P 65a68ee1eb024af699fad9e16b47c5c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:59:40.076148 13351 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65a68ee1eb024af699fad9e16b47c5c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:59:40.077126 13351 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65a68ee1eb024af699fad9e16b47c5c0 [term 1 LEADER]: Becoming Leader. State: Replica: 65a68ee1eb024af699fad9e16b47c5c0, State: Running, Role: LEADER
I20260928 14:59:40.077715 13351 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65a68ee1eb024af699fad9e16b47c5c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a68ee1eb024af699fad9e16b47c5c0" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34493 } }
I20260928 14:59:40.078421 13346 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65a68ee1eb024af699fad9e16b47c5c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:59:40.080701 13353 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65a68ee1eb024af699fad9e16b47c5c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65a68ee1eb024af699fad9e16b47c5c0. Latest consensus state: current_term: 1 leader_uuid: "65a68ee1eb024af699fad9e16b47c5c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a68ee1eb024af699fad9e16b47c5c0" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34493 } } }
I20260928 14:59:40.081190 13353 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65a68ee1eb024af699fad9e16b47c5c0 [sys.catalog]: This master's current role is: LEADER
I20260928 14:59:40.081521 13352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65a68ee1eb024af699fad9e16b47c5c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65a68ee1eb024af699fad9e16b47c5c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a68ee1eb024af699fad9e16b47c5c0" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34493 } } }
I20260928 14:59:40.088150 13352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65a68ee1eb024af699fad9e16b47c5c0 [sys.catalog]: This master's current role is: LEADER
I20260928 14:59:40.089496 13358 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:59:40.094272 13358 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:59:40.103092 13358 catalog_manager.cc:1347] Generated new cluster ID: cb4858c6e60140729d41551569c3553b
I20260928 14:59:40.103211 13358 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:59:40.116118 13358 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:59:40.118088 13358 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:59:40.128403 13358 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65a68ee1eb024af699fad9e16b47c5c0: Generated new TSK 0
I20260928 14:59:40.129297 13358 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:59:40.186086 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 13279
I20260928 14:59:40.187346 13341 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:59:40.287074 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 13279
2026-09-28T14:59:40Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2091 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:59:40 dist-test-slave-j968 krb5kdc[13374](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:59:40 dist-test-slave-j968 krb5kdc[13374](info): set up 2 sockets
Sep 28 14:59:40 dist-test-slave-j968 krb5kdc[13374](info): commencing operation
krb5kdc: starting...
W20260928 14:59:41.397370 31904 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Sep 28 14:59:41 dist-test-slave-j968 krb5kdc[13374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:59:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:59:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab.
Entry for principal kudu/127.31.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab.
I20260928 14:59:41.643689 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:43875
--webserver_interface=127.31.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:39673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.40.62:43875
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:59:41.993809 13387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:59:41.994136 13387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:59:41.994227 13387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:59:42.001691 13387 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:59:42.001830 13387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:59:42.001886 13387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:59:42.001942 13387 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:59:42.001986 13387 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:59:42.012373 13387 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:39673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.40.62:43875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:43875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:59:42.013731 13387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:59:42.015050 13387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:59:42.027173 13392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:59:42.027285 13395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:59:42.027321 13393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:59:42.028899 13387 server_base.cc:1034] running on GCE node
I20260928 14:59:42.029633 13387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:59:42.031055 13387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:59:42.032260 13387 hybrid_clock.cc:648] HybridClock initialized: now 1790607582032207 us; error 39 us; skew 500 ppm
I20260928 14:59:42.032768 13387 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:59:42 dist-test-slave-j968 krb5kdc[13374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607582, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:59:42.036088 13387 init.cc:377] Logged in from keytab as kudu/127.31.40.62@KRBTEST.COM (short username kudu)
I20260928 14:59:42.037664 13387 webserver.cc:460] Webserver started at http://127.31.40.62:36845/ using document root <none> and password file <none>
I20260928 14:59:42.038614 13387 fs_manager.cc:359] Metadata directory not provided
I20260928 14:59:42.038843 13387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:59:42.039222 13387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:59:42.042629 13387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/master-0/data/instance:
uuid: "40864fbc798f4c4b8033a199f800a219"
format_stamp: "Formatted at 2026-09-28 14:59:42 on dist-test-slave-j968"
server_key: "463f55961d213e59e4bd0704c14b73f7"
server_key_iv: "527ed252030efd282a04fd5f1b04591d"
server_key_version: "encryptionkey@0"
I20260928 14:59:42.043479 13387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/master-0/wal/instance:
uuid: "40864fbc798f4c4b8033a199f800a219"
format_stamp: "Formatted at 2026-09-28 14:59:42 on dist-test-slave-j968"
server_key: "463f55961d213e59e4bd0704c14b73f7"
server_key_iv: "527ed252030efd282a04fd5f1b04591d"
server_key_version: "encryptionkey@0"
I20260928 14:59:42.049182 13387 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 14:59:42.053122 13402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:59:42.054596 13387 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260928 14:59:42.054807 13387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/master-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/master-0/wal
uuid: "40864fbc798f4c4b8033a199f800a219"
format_stamp: "Formatted at 2026-09-28 14:59:42 on dist-test-slave-j968"
server_key: "463f55961d213e59e4bd0704c14b73f7"
server_key_iv: "527ed252030efd282a04fd5f1b04591d"
server_key_version: "encryptionkey@0"
I20260928 14:59:42.055038 13387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:59:42.072870 13387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:59:42.077406 13387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:59:42.077732 13387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:59:42.097124 13387 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.62:43875
I20260928 14:59:42.097172 13453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.62:43875 every 8 connection(s)
I20260928 14:59:42.099049 13387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
I20260928 14:59:42.105050 13454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:59:42.108528 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 13387
I20260928 14:59:42.108747 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/master-0/wal/instance
I20260928 14:59:42.109421 31904 external_mini_cluster.cc:1428] Setting key 6c157fbc370b1473ce972d2eeb6159dd
I20260928 14:59:42.118813 13454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219: Bootstrap starting.
I20260928 14:59:42.122437 13454 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219: Neither blocks nor log segments found. Creating new log.
I20260928 14:59:42.123636 13454 log.cc:824] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219: Log is configured to *not* fsync() on all Append() calls
Sep 28 14:59:42 dist-test-slave-j968 krb5kdc[13374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:59:42.127312 13454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219: No bootstrap required, opened a new log
I20260928 14:59:42.136101 13457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:42.111808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55238 (local address 127.31.40.62:43875)
0928 14:59:42.112848 (+ 1040us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:42.112862 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:42.112909 (+ 47us) server_negotiation.cc:407] Connection header received
0928 14:59:42.114942 (+ 2033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:42.114969 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:42.115493 (+ 524us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:42.116023 (+ 530us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:59:42.117706 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:42.119277 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:42.120842 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:42.121616 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:42.121866 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:42.127045 (+ 5179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:59:42.127055 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:59:42.127071 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:59:42.127110 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:59:42.130537 (+ 3427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:42.131329 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:42.131337 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:42.131343 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:42.131513 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:42.132107 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:42.132112 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:42.132120 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:42.132712 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:59:42.133089 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:42.133547 (+ 458us) server_negotiation.cc:299] Negotiation successful
0928 14:59:42.134279 (+ 732us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":778,"thread_start_us":469,"threads_started":1}
I20260928 14:59:42.139431 13454 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40864fbc798f4c4b8033a199f800a219" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 43875 } }
I20260928 14:59:42.140012 13454 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:59:42.140131 13454 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40864fbc798f4c4b8033a199f800a219, State: Initialized, Role: FOLLOWER
I20260928 14:59:42.140862 13454 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40864fbc798f4c4b8033a199f800a219" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 43875 } }
I20260928 14:59:42.141170 13454 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:59:42.141322 13454 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:59:42.141527 13454 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:59:42.145082 13454 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40864fbc798f4c4b8033a199f800a219" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 43875 } }
I20260928 14:59:42.145726 13454 leader_election.cc:302] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40864fbc798f4c4b8033a199f800a219; no voters:
I20260928 14:59:42.146576 13454 leader_election.cc:288] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:59:42.146682 13459 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:59:42.147539 13459 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219 [term 1 LEADER]: Becoming Leader. State: Replica: 40864fbc798f4c4b8033a199f800a219, State: Running, Role: LEADER
I20260928 14:59:42.148151 13459 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40864fbc798f4c4b8033a199f800a219" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 43875 } }
I20260928 14:59:42.148993 13454 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:59:42.151273 13461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40864fbc798f4c4b8033a199f800a219. Latest consensus state: current_term: 1 leader_uuid: "40864fbc798f4c4b8033a199f800a219" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40864fbc798f4c4b8033a199f800a219" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 43875 } } }
I20260928 14:59:42.151834 13461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219 [sys.catalog]: This master's current role is: LEADER
I20260928 14:59:42.155020 13460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40864fbc798f4c4b8033a199f800a219" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40864fbc798f4c4b8033a199f800a219" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 43875 } } }
I20260928 14:59:42.155519 13460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219 [sys.catalog]: This master's current role is: LEADER
I20260928 14:59:42.156970 13464 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:59:42.162611 13464 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:59:42.172058 13464 catalog_manager.cc:1347] Generated new cluster ID: dd0d8e389e114f4fbaf6eb4039ae60a7
I20260928 14:59:42.172215 13464 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:59:42.190263 13464 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:59:42.191743 13464 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:59:42.200078 13464 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219: Generated new TSK 0
I20260928 14:59:42.201205 13464 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.40.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab.
Entry for principal kudu/127.31.40.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab.
I20260928 14:59:42.271925 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.1:0
--local_ip_for_outbound_sockets=127.31.40.1
--webserver_interface=127.31.40.1
--webserver_port=0
--tserver_master_addrs=127.31.40.62:43875
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab
--principal=kudu/127.31.40.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:39673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:59:42.620121 13480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:59:42.620416 13480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:59:42.620594 13480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:59:42.628136 13480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:59:42.628253 13480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:59:42.628451 13480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.1
I20260928 14:59:42.638273 13480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:39673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab
--principal=kudu/127.31.40.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.40.1
--webserver_port=0
--tserver_master_addrs=127.31.40.62:43875
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.1
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:59:42.639694 13480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:59:42.641144 13480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:59:42.654985 13485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:59:42.654904 13488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:59:42.655018 13486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:59:42.655954 13480 server_base.cc:1034] running on GCE node
I20260928 14:59:42.656605 13480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:59:42.657672 13480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:59:42.658874 13480 hybrid_clock.cc:648] HybridClock initialized: now 1790607582658813 us; error 56 us; skew 500 ppm
I20260928 14:59:42.659258 13480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:59:42 dist-test-slave-j968 krb5kdc[13374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607582, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:59:42.662729 13480 init.cc:377] Logged in from keytab as kudu/127.31.40.1@KRBTEST.COM (short username kudu)
I20260928 14:59:42.664391 13480 webserver.cc:460] Webserver started at http://127.31.40.1:36631/ using document root <none> and password file <none>
I20260928 14:59:42.665092 13480 fs_manager.cc:359] Metadata directory not provided
I20260928 14:59:42.665197 13480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:59:42.665472 13480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:59:42.668638 13480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-0/data/instance:
uuid: "99586f213ec449ef8f1a8b1255dbb372"
format_stamp: "Formatted at 2026-09-28 14:59:42 on dist-test-slave-j968"
server_key: "b0b4b056fd8add989e3eb7e289374c91"
server_key_iv: "a9788ed67f36beaa5d647573f7b9802c"
server_key_version: "encryptionkey@0"
I20260928 14:59:42.669379 13480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-0/wal/instance:
uuid: "99586f213ec449ef8f1a8b1255dbb372"
format_stamp: "Formatted at 2026-09-28 14:59:42 on dist-test-slave-j968"
server_key: "b0b4b056fd8add989e3eb7e289374c91"
server_key_iv: "a9788ed67f36beaa5d647573f7b9802c"
server_key_version: "encryptionkey@0"
I20260928 14:59:42.674932 13480 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 14:59:42.678908 13495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:59:42.680588 13480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:59:42.680794 13480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-0/wal
uuid: "99586f213ec449ef8f1a8b1255dbb372"
format_stamp: "Formatted at 2026-09-28 14:59:42 on dist-test-slave-j968"
server_key: "b0b4b056fd8add989e3eb7e289374c91"
server_key_iv: "a9788ed67f36beaa5d647573f7b9802c"
server_key_version: "encryptionkey@0"
I20260928 14:59:42.681039 13480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:59:42.697018 13480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:59:42.701805 13480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:59:42.702174 13480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:59:42.703161 13480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:59:42.704604 13480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:59:42.704715 13480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:59:42.704905 13480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:59:42.704958 13480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:59:42.739508 13480 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.1:45251
I20260928 14:59:42.739545 13607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.1:45251 every 8 connection(s)
I20260928 14:59:42.740991 13480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
I20260928 14:59:42.749614 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 13480
I20260928 14:59:42.749954 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-0/wal/instance
I20260928 14:59:42.750847 31904 external_mini_cluster.cc:1428] Setting key 9a9e9a7cd7a0f7b2b4149dc8a31d66bb
Sep 28 14:59:42 dist-test-slave-j968 krb5kdc[13374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607582, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.1@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:59:42.762840 13610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:42.743964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:48487 (local address 127.31.40.62:43875)
0928 14:59:42.744472 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:42.744476 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:42.745335 (+ 859us) server_negotiation.cc:407] Connection header received
0928 14:59:42.747196 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:42.747202 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:42.747298 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:42.747431 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:59:42.749082 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:42.750019 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:42.752319 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:42.752748 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:42.752903 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:42.756670 (+ 3767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:59:42.756675 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:59:42.756679 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:59:42.756712 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:59:42.759442 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:42.760335 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:42.760341 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:42.760345 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:42.760515 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:42.760970 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:42.760975 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:42.760979 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:42.761271 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:59:42.761439 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:42.762223 (+ 784us) server_negotiation.cc:299] Negotiation successful
0928 14:59:42.762472 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":263,"threads_started":1}
I20260928 14:59:42.763934 13611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:42.744420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.1:48487)
0928 14:59:42.745162 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:42.745207 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 14:59:42.746908 (+ 1701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:59:42.747676 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:59:42.747684 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:59:42.748236 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:59:42.748905 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:42.748925 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:42.750251 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:42.750260 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:42.752142 (+ 1882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:42.752156 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:42.752999 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:42.753010 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:42.753271 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:42.754021 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:59:42.754055 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:59:42.756420 (+ 2365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:59:42.759608 (+ 3188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:59:42.759619 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:59:42.759640 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:59:42.760196 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:59:42.760628 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:59:42.760634 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:59:42.760637 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:59:42.760836 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:59:42.761420 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:59:42.761428 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:59:42.761816 (+ 388us) client_negotiation.cc:769] Sending connection context
0928 14:59:42.762118 (+ 302us) client_negotiation.cc:240] Negotiation successful
0928 14:59:42.762704 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":279,"threads_started":1}
I20260928 14:59:42.765635 13608 heartbeater.cc:347] Connected to a master server at 127.31.40.62:43875
I20260928 14:59:42.765983 13608 heartbeater.cc:464] Registering TS with master...
I20260928 14:59:42.766949 13608 heartbeater.cc:511] Master 127.31.40.62:43875 requested a full tablet report, sending...
I20260928 14:59:42.769687 13419 ts_manager.cc:194] Registered new tserver with Master: 99586f213ec449ef8f1a8b1255dbb372 (127.31.40.1:45251)
I20260928 14:59:42.772044 13419 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.1@KRBTEST.COM'} at 127.31.40.1:48487
WARNING: no policy specified for kudu/127.31.40.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab.
Entry for principal kudu/127.31.40.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab.
I20260928 14:59:42.808148 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.2:0
--local_ip_for_outbound_sockets=127.31.40.2
--webserver_interface=127.31.40.2
--webserver_port=0
--tserver_master_addrs=127.31.40.62:43875
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab
--principal=kudu/127.31.40.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:39673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:59:43.143858 13614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:59:43.144233 13614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:59:43.144333 13614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:59:43.152148 13614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:59:43.152273 13614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:59:43.152390 13614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.2
I20260928 14:59:43.162814 13614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:39673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab
--principal=kudu/127.31.40.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.40.2
--webserver_port=0
--tserver_master_addrs=127.31.40.62:43875
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.2
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:59:43.164181 13614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:59:43.165455 13614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:59:43.178864 13619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:59:43.178969 13622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:59:43.178936 13620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:59:43.180912 13614 server_base.cc:1034] running on GCE node
I20260928 14:59:43.181733 13614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:59:43.182942 13614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:59:43.184193 13614 hybrid_clock.cc:648] HybridClock initialized: now 1790607583184170 us; error 57 us; skew 500 ppm
I20260928 14:59:43.184545 13614 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:59:43 dist-test-slave-j968 krb5kdc[13374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607583, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:59:43.188023 13614 init.cc:377] Logged in from keytab as kudu/127.31.40.2@KRBTEST.COM (short username kudu)
I20260928 14:59:43.189554 13614 webserver.cc:460] Webserver started at http://127.31.40.2:39369/ using document root <none> and password file <none>
I20260928 14:59:43.190292 13614 fs_manager.cc:359] Metadata directory not provided
I20260928 14:59:43.190448 13614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:59:43.190773 13614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:59:43.193936 13614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-1/data/instance:
uuid: "c34083cb0fd441ec933ad27e4602ee5e"
format_stamp: "Formatted at 2026-09-28 14:59:43 on dist-test-slave-j968"
server_key: "21ed01889dc007b0666c606660d5b18e"
server_key_iv: "7d60ad8a36b4fedaac4945143323055b"
server_key_version: "encryptionkey@0"
I20260928 14:59:43.194739 13614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-1/wal/instance:
uuid: "c34083cb0fd441ec933ad27e4602ee5e"
format_stamp: "Formatted at 2026-09-28 14:59:43 on dist-test-slave-j968"
server_key: "21ed01889dc007b0666c606660d5b18e"
server_key_iv: "7d60ad8a36b4fedaac4945143323055b"
server_key_version: "encryptionkey@0"
I20260928 14:59:43.200352 13614 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 14:59:43.204325 13629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:59:43.206161 13614 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 14:59:43.206406 13614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-1/wal
uuid: "c34083cb0fd441ec933ad27e4602ee5e"
format_stamp: "Formatted at 2026-09-28 14:59:43 on dist-test-slave-j968"
server_key: "21ed01889dc007b0666c606660d5b18e"
server_key_iv: "7d60ad8a36b4fedaac4945143323055b"
server_key_version: "encryptionkey@0"
I20260928 14:59:43.206656 13614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:59:43.226801 13614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:59:43.231225 13614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:59:43.231557 13614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:59:43.232677 13614 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:59:43.234203 13614 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:59:43.234315 13614 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:59:43.234467 13614 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:59:43.234539 13614 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:59:43.270057 13614 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.2:38411
I20260928 14:59:43.270105 13741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.2:38411 every 8 connection(s)
I20260928 14:59:43.271365 13614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
I20260928 14:59:43.272982 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 13614
I20260928 14:59:43.273175 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-1/wal/instance
I20260928 14:59:43.273619 31904 external_mini_cluster.cc:1428] Setting key 0bc72ba2b7ea2d9a4c464a4c4aff9ba4
Sep 28 14:59:43 dist-test-slave-j968 krb5kdc[13374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607583, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.2@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:59:43.296977 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:43.275481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:42459 (local address 127.31.40.62:43875)
0928 14:59:43.275996 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:43.276002 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:43.277183 (+ 1181us) server_negotiation.cc:407] Connection header received
0928 14:59:43.278968 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:43.278976 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:43.279066 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:43.279185 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:59:43.280858 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:43.282066 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:43.283782 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:43.284620 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:43.284771 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:43.289208 (+ 4437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:59:43.289218 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:59:43.289225 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:59:43.289276 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:59:43.293301 (+ 4025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:43.294316 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:43.294321 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:43.294323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:43.294467 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:43.294962 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:43.294972 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:43.294975 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:43.295286 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:59:43.295455 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:43.296321 (+ 866us) server_negotiation.cc:299] Negotiation successful
0928 14:59:43.296561 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":242,"threads_started":1}
WARNING: no policy specified for kudu/127.31.40.3@KRBTEST.COM; defaulting to no policy
I20260928 14:59:43.298267 13745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:43.276154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.2:42459)
0928 14:59:43.277002 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:43.277049 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 14:59:43.278691 (+ 1642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:59:43.279403 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:59:43.279412 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:59:43.280078 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:59:43.280676 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:43.280694 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:43.282274 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:43.282280 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:43.283607 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:43.283621 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:43.284867 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:43.284872 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:43.285163 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:43.286141 (+ 978us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:59:43.286174 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:59:43.288901 (+ 2727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:59:43.293532 (+ 4631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:59:43.293541 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:59:43.293558 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:59:43.294110 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:59:43.294599 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:59:43.294604 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:59:43.294607 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:59:43.294825 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:59:43.295437 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:59:43.295446 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:59:43.295853 (+ 407us) client_negotiation.cc:769] Sending connection context
0928 14:59:43.296221 (+ 368us) client_negotiation.cc:240] Negotiation successful
0928 14:59:43.296834 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":377,"threads_started":1}
I20260928 14:59:43.299960 13742 heartbeater.cc:347] Connected to a master server at 127.31.40.62:43875
I20260928 14:59:43.300401 13742 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.3@KRBTEST.COM" created.
I20260928 14:59:43.301373 13742 heartbeater.cc:511] Master 127.31.40.62:43875 requested a full tablet report, sending...
I20260928 14:59:43.303158 13419 ts_manager.cc:194] Registered new tserver with Master: c34083cb0fd441ec933ad27e4602ee5e (127.31.40.2:38411)
I20260928 14:59:43.304194 13419 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.2@KRBTEST.COM'} at 127.31.40.2:42459
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab.
Entry for principal kudu/127.31.40.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab.
I20260928 14:59:43.331723 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.3:0
--local_ip_for_outbound_sockets=127.31.40.3
--webserver_interface=127.31.40.3
--webserver_port=0
--tserver_master_addrs=127.31.40.62:43875
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab
--principal=kudu/127.31.40.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:39673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:59:43.670832 13748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:59:43.671128 13748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:59:43.671298 13748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:59:43.678671 13748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:59:43.678786 13748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:59:43.678979 13748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.3
I20260928 14:59:43.689122 13748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:39673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab
--principal=kudu/127.31.40.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.40.3
--webserver_port=0
--tserver_master_addrs=127.31.40.62:43875
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.3
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:59:43.690500 13748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:59:43.691969 13748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:59:43.706310 13753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:59:43.706274 13756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:59:43.706358 13754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:59:43.707496 13748 server_base.cc:1034] running on GCE node
I20260928 14:59:43.708120 13748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:59:43.709029 13748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:59:43.710232 13748 hybrid_clock.cc:648] HybridClock initialized: now 1790607583710164 us; error 56 us; skew 500 ppm
I20260928 14:59:43.710621 13748 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:59:43 dist-test-slave-j968 krb5kdc[13374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607583, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:59:43.713814 13748 init.cc:377] Logged in from keytab as kudu/127.31.40.3@KRBTEST.COM (short username kudu)
I20260928 14:59:43.715236 13748 webserver.cc:460] Webserver started at http://127.31.40.3:43059/ using document root <none> and password file <none>
I20260928 14:59:43.715932 13748 fs_manager.cc:359] Metadata directory not provided
I20260928 14:59:43.716053 13748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:59:43.716306 13748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:59:43.719195 13748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-2/data/instance:
uuid: "510a02d078ff4965b5dfee7317012028"
format_stamp: "Formatted at 2026-09-28 14:59:43 on dist-test-slave-j968"
server_key: "67992f9ae5900758844c16c37b5a00d6"
server_key_iv: "194cf94a9a7262c84a9b9ae52218bd35"
server_key_version: "encryptionkey@0"
I20260928 14:59:43.720016 13748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-2/wal/instance:
uuid: "510a02d078ff4965b5dfee7317012028"
format_stamp: "Formatted at 2026-09-28 14:59:43 on dist-test-slave-j968"
server_key: "67992f9ae5900758844c16c37b5a00d6"
server_key_iv: "194cf94a9a7262c84a9b9ae52218bd35"
server_key_version: "encryptionkey@0"
I20260928 14:59:43.725276 13748 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260928 14:59:43.728868 13763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:59:43.730286 13748 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:59:43.730427 13748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-2/wal
uuid: "510a02d078ff4965b5dfee7317012028"
format_stamp: "Formatted at 2026-09-28 14:59:43 on dist-test-slave-j968"
server_key: "67992f9ae5900758844c16c37b5a00d6"
server_key_iv: "194cf94a9a7262c84a9b9ae52218bd35"
server_key_version: "encryptionkey@0"
I20260928 14:59:43.730597 13748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:59:43.743785 13748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:59:43.748539 13748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:59:43.748890 13748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:59:43.749907 13748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:59:43.751219 13748 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:59:43.751291 13748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:59:43.751374 13748 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:59:43.751427 13748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:59:43.776436 13608 heartbeater.cc:503] Master 127.31.40.62:43875 was elected leader, sending a full tablet report...
I20260928 14:59:43.785877 13748 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.3:39831
I20260928 14:59:43.786000 13875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.3:39831 every 8 connection(s)
I20260928 14:59:43.787249 13748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
I20260928 14:59:43.796661 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 13748
I20260928 14:59:43.796888 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-2/wal/instance
I20260928 14:59:43.797396 31904 external_mini_cluster.cc:1428] Setting key 4db305b0cfba2d72ae663ce951702afc
Sep 28 14:59:43 dist-test-slave-j968 krb5kdc[13374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607583, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.3@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:59:43.807834 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:43.790017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:55427 (local address 127.31.40.62:43875)
0928 14:59:43.790237 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:43.790241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:43.791433 (+ 1192us) server_negotiation.cc:407] Connection header received
0928 14:59:43.793331 (+ 1898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:43.793337 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:43.793442 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:43.793554 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:59:43.795121 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:43.795934 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:43.797717 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:43.798288 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:43.798427 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:43.802301 (+ 3874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:59:43.802307 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:59:43.802310 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:59:43.802343 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:59:43.804773 (+ 2430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:43.805531 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:43.805537 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:43.805541 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:43.805663 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:43.806095 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:43.806101 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:43.806105 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:43.806392 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:59:43.806554 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:43.807269 (+ 715us) server_negotiation.cc:299] Negotiation successful
0928 14:59:43.807500 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 14:59:43.808878 13878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:43.790528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.3:55427)
0928 14:59:43.791255 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:43.791301 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 14:59:43.793062 (+ 1761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:59:43.793787 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:59:43.793796 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:59:43.794365 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:59:43.794947 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:43.794966 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:43.796102 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:43.796106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:43.797568 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:43.797580 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:43.798525 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:43.798532 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:43.798847 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:43.799633 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:59:43.799655 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:59:43.802076 (+ 2421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:59:43.804917 (+ 2841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:59:43.804927 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:59:43.804944 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:59:43.805371 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:59:43.805779 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:59:43.805784 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:59:43.805787 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:59:43.805978 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:59:43.806534 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:59:43.806542 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:59:43.806883 (+ 341us) client_negotiation.cc:769] Sending connection context
0928 14:59:43.807177 (+ 294us) client_negotiation.cc:240] Negotiation successful
0928 14:59:43.807736 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":288,"threads_started":1}
I20260928 14:59:43.810343 13876 heartbeater.cc:347] Connected to a master server at 127.31.40.62:43875
I20260928 14:59:43.810649 13876 heartbeater.cc:464] Registering TS with master...
I20260928 14:59:43.811398 13876 heartbeater.cc:511] Master 127.31.40.62:43875 requested a full tablet report, sending...
I20260928 14:59:43.813054 13419 ts_manager.cc:194] Registered new tserver with Master: 510a02d078ff4965b5dfee7317012028 (127.31.40.3:39831)
I20260928 14:59:43.813993 31904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:59:43.814200 13419 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.3@KRBTEST.COM'} at 127.31.40.3:55427
I20260928 14:59:43.832233 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:43.818805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55248 (local address 127.31.40.62:43875)
0928 14:59:43.819013 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:43.819017 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:43.819462 (+ 445us) server_negotiation.cc:407] Connection header received
0928 14:59:43.819582 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:43.819586 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:43.819660 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:43.819747 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:59:43.821305 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:43.822061 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:43.823524 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:43.824109 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:43.824219 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:43.826652 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:59:43.826657 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:59:43.826660 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:59:43.826694 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:59:43.829398 (+ 2704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:43.830164 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:43.830169 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:43.830171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:43.830293 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:43.830721 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:43.830725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:43.830727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:43.831132 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:59:43.831330 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:43.831716 (+ 386us) server_negotiation.cc:299] Negotiation successful
0928 14:59:43.831987 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 14:59:43.836683 13419 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55248:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:59:43.843855 13419 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:59:43.881004 13890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:43.856782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.2:38411 (local address 127.0.0.1:57630)
0928 14:59:43.858058 (+ 1276us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:43.858101 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 14:59:43.858299 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:59:43.873793 (+ 15494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:59:43.873798 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:59:43.873835 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:59:43.874638 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:43.874652 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:43.876642 (+ 1990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:43.876647 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:43.878544 (+ 1897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:43.878556 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:43.880218 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:43.880224 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:43.880415 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:43.880458 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 14:59:43.880551 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 14:59:43.880684 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":975,"mutex_wait_us":69,"thread_start_us":15072,"threads_started":1}
I20260928 14:59:43.882081 13889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:43.856950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57630 (local address 127.31.40.2:38411)
0928 14:59:43.872851 (+ 15901us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:43.872861 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:43.872895 (+ 34us) server_negotiation.cc:407] Connection header received
0928 14:59:43.873006 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:43.873015 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:43.873336 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:43.874507 (+ 1171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:59:43.874835 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:43.876486 (+ 1651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:43.878711 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:43.880012 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:43.881515 (+ 1503us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:43.881651 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:43.881712 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 14:59:43.881849 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15718,"thread_start_us":16549,"threads_started":1}
I20260928 14:59:43.882768 13892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:43.858881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.3:39831 (local address 127.0.0.1:36978)
0928 14:59:43.872858 (+ 13977us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:43.872877 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 14:59:43.873012 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:59:43.873787 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:59:43.873796 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:59:43.873867 (+ 71us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:59:43.874565 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:43.874581 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:43.876710 (+ 2129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:43.876716 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:43.879523 (+ 2807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:43.879534 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:43.881715 (+ 2181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:43.881721 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:43.882019 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:43.882155 (+ 136us) client_negotiation.cc:769] Sending connection context
0928 14:59:43.882243 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 14:59:43.882383 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13866,"mutex_wait_us":348,"spinlock_wait_cycles":4992,"thread_start_us":13696,"threads_started":1}
I20260928 14:59:43.882863 13891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:43.857869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41224 (local address 127.31.40.1:45251)
0928 14:59:43.873631 (+ 15762us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:43.873640 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:43.873666 (+ 26us) server_negotiation.cc:407] Connection header received
0928 14:59:43.873760 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:43.873767 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:43.874016 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:43.874221 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:59:43.874979 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:43.876589 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:43.879711 (+ 3122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:43.881181 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:43.881304 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:43.881545 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:43.882352 (+ 807us) server_negotiation.cc:299] Negotiation successful
0928 14:59:43.882527 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15515,"spinlock_wait_cycles":28928,"thread_start_us":15492,"threads_started":1}
I20260928 14:59:43.882928 13893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:43.859272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36978 (local address 127.31.40.3:39831)
0928 14:59:43.871727 (+ 12455us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:43.871742 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:43.873168 (+ 1426us) server_negotiation.cc:407] Connection header received
0928 14:59:43.873281 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:43.873288 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:43.873543 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:43.873784 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:59:43.874787 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:43.876498 (+ 1711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:43.879710 (+ 3212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:43.881181 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:43.881304 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:43.881544 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:43.882447 (+ 903us) server_negotiation.cc:299] Negotiation successful
0928 14:59:43.882604 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12198,"thread_start_us":12473,"threads_started":1}
I20260928 14:59:43.886132 13677 tablet_service.cc:1467] Processing CreateTablet for tablet 46f0e1d98e694e47ad18ea1cd309c1b9 (DEFAULT_TABLE table=test-table [id=97c1c120da104af6bd2ed3875c69979b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:59:43.886514 13811 tablet_service.cc:1467] Processing CreateTablet for tablet 46f0e1d98e694e47ad18ea1cd309c1b9 (DEFAULT_TABLE table=test-table [id=97c1c120da104af6bd2ed3875c69979b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:59:43.887416 13677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46f0e1d98e694e47ad18ea1cd309c1b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:59:43.887744 13811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46f0e1d98e694e47ad18ea1cd309c1b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:59:43.895787 13888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:43.857197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:45251 (local address 127.0.0.1:41224)
0928 14:59:43.871444 (+ 14247us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:43.871478 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 14:59:43.871697 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:59:43.874224 (+ 2527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:59:43.874229 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:59:43.874252 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:59:43.874759 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:43.874771 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:43.876736 (+ 1965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:43.876742 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:43.879519 (+ 2777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:43.879534 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:43.881726 (+ 2192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:43.881732 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:43.882017 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:43.882156 (+ 139us) client_negotiation.cc:769] Sending connection context
0928 14:59:43.882243 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 14:59:43.895313 (+ 13070us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14024,"mutex_wait_us":998,"thread_start_us":794,"threads_started":1}
I20260928 14:59:43.899417 13894 tablet_bootstrap.cc:492] T 46f0e1d98e694e47ad18ea1cd309c1b9 P c34083cb0fd441ec933ad27e4602ee5e: Bootstrap starting.
I20260928 14:59:43.900408 13543 tablet_service.cc:1467] Processing CreateTablet for tablet 46f0e1d98e694e47ad18ea1cd309c1b9 (DEFAULT_TABLE table=test-table [id=97c1c120da104af6bd2ed3875c69979b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:59:43.901949 13543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46f0e1d98e694e47ad18ea1cd309c1b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:59:43.904299 13894 tablet_bootstrap.cc:654] T 46f0e1d98e694e47ad18ea1cd309c1b9 P c34083cb0fd441ec933ad27e4602ee5e: Neither blocks nor log segments found. Creating new log.
I20260928 14:59:43.904645 13895 tablet_bootstrap.cc:492] T 46f0e1d98e694e47ad18ea1cd309c1b9 P 510a02d078ff4965b5dfee7317012028: Bootstrap starting.
I20260928 14:59:43.905695 13894 log.cc:824] T 46f0e1d98e694e47ad18ea1cd309c1b9 P c34083cb0fd441ec933ad27e4602ee5e: Log is configured to *not* fsync() on all Append() calls
I20260928 14:59:43.908366 13895 tablet_bootstrap.cc:654] T 46f0e1d98e694e47ad18ea1cd309c1b9 P 510a02d078ff4965b5dfee7317012028: Neither blocks nor log segments found. Creating new log.
I20260928 14:59:43.908674 13894 tablet_bootstrap.cc:492] T 46f0e1d98e694e47ad18ea1cd309c1b9 P c34083cb0fd441ec933ad27e4602ee5e: No bootstrap required, opened a new log
I20260928 14:59:43.908960 13894 ts_tablet_manager.cc:1397] T 46f0e1d98e694e47ad18ea1cd309c1b9 P c34083cb0fd441ec933ad27e4602ee5e: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260928 14:59:43.909698 13895 log.cc:824] T 46f0e1d98e694e47ad18ea1cd309c1b9 P 510a02d078ff4965b5dfee7317012028: Log is configured to *not* fsync() on all Append() calls
I20260928 14:59:43.913219 13895 tablet_bootstrap.cc:492] T 46f0e1d98e694e47ad18ea1cd309c1b9 P 510a02d078ff4965b5dfee7317012028: No bootstrap required, opened a new log
I20260928 14:59:43.913517 13895 ts_tablet_manager.cc:1397] T 46f0e1d98e694e47ad18ea1cd309c1b9 P 510a02d078ff4965b5dfee7317012028: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260928 14:59:43.917681 13894 raft_consensus.cc:348] T 46f0e1d98e694e47ad18ea1cd309c1b9 P c34083cb0fd441ec933ad27e4602ee5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c34083cb0fd441ec933ad27e4602ee5e" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 38411 } } peers { permanent_uuid: "99586f213ec449ef8f1a8b1255dbb372" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 45251 } } peers { permanent_uuid: "510a02d078ff4965b5dfee7317012028" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 39831 } }
I20260928 14:59:43.918344 13894 raft_consensus.cc:374] T 46f0e1d98e694e47ad18ea1cd309c1b9 P c34083cb0fd441ec933ad27e4602ee5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:59:43.918519 13894 raft_consensus.cc:729] T 46f0e1d98e694e47ad18ea1cd309c1b9 P c34083cb0fd441ec933ad27e4602ee5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c34083cb0fd441ec933ad27e4602ee5e, State: Initialized, Role: FOLLOWER
I20260928 14:59:43.918648 13898 tablet_bootstrap.cc:492] T 46f0e1d98e694e47ad18ea1cd309c1b9 P 99586f213ec449ef8f1a8b1255dbb372: Bootstrap starting.
I20260928 14:59:43.919416 13894 consensus_queue.cc:260] T 46f0e1d98e694e47ad18ea1cd309c1b9 P c34083cb0fd441ec933ad27e4602ee5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c34083cb0fd441ec933ad27e4602ee5e" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 38411 } } peers { permanent_uuid: "99586f213ec449ef8f1a8b1255dbb372" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 45251 } } peers { permanent_uuid: "510a02d078ff4965b5dfee7317012028" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 39831 } }
I20260928 14:59:43.920588 13742 heartbeater.cc:503] Master 127.31.40.62:43875 was elected leader, sending a full tablet report...
I20260928 14:59:43.921495 13894 ts_tablet_manager.cc:1428] T 46f0e1d98e694e47ad18ea1cd309c1b9 P c34083cb0fd441ec933ad27e4602ee5e: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
I20260928 14:59:43.923063 13898 tablet_bootstrap.cc:654] T 46f0e1d98e694e47ad18ea1cd309c1b9 P 99586f213ec449ef8f1a8b1255dbb372: Neither blocks nor log segments found. Creating new log.
I20260928 14:59:43.924436 13898 log.cc:824] T 46f0e1d98e694e47ad18ea1cd309c1b9 P 99586f213ec449ef8f1a8b1255dbb372: Log is configured to *not* fsync() on all Append() calls
I20260928 14:59:43.924317 13895 raft_consensus.cc:348] T 46f0e1d98e694e47ad18ea1cd309c1b9 P 510a02d078ff4965b5dfee7317012028 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c34083cb0fd441ec933ad27e4602ee5e" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 38411 } } peers { permanent_uuid: "99586f213ec449ef8f1a8b1255dbb372" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 45251 } } peers { permanent_uuid: "510a02d078ff4965b5dfee7317012028" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 39831 } }
I20260928 14:59:43.924811 13895 raft_consensus.cc:374] T 46f0e1d98e694e47ad18ea1cd309c1b9 P 510a02d078ff4965b5dfee7317012028 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:59:43.924979 13895 raft_consensus.cc:729] T 46f0e1d98e694e47ad18ea1cd309c1b9 P 510a02d078ff4965b5dfee7317012028 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 510a02d078ff4965b5dfee7317012028, State: Initialized, Role: FOLLOWER
I20260928 14:59:43.925685 13895 consensus_queue.cc:260] T 46f0e1d98e694e47ad18ea1cd309c1b9 P 510a02d078ff4965b5dfee7317012028 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c34083cb0fd441ec933ad27e4602ee5e" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 38411 } } peers { permanent_uuid: "99586f213ec449ef8f1a8b1255dbb372" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 45251 } } peers { permanent_uuid: "510a02d078ff4965b5dfee7317012028" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 39831 } }
I20260928 14:59:43.926815 13876 heartbeater.cc:503] Master 127.31.40.62:43875 was elected leader, sending a full tablet report...
I20260928 14:59:43.927651 13895 ts_tablet_manager.cc:1428] T 46f0e1d98e694e47ad18ea1cd309c1b9 P 510a02d078ff4965b5dfee7317012028: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260928 14:59:43.927798 13898 tablet_bootstrap.cc:492] T 46f0e1d98e694e47ad18ea1cd309c1b9 P 99586f213ec449ef8f1a8b1255dbb372: No bootstrap required, opened a new log
I20260928 14:59:43.928124 13898 ts_tablet_manager.cc:1397] T 46f0e1d98e694e47ad18ea1cd309c1b9 P 99586f213ec449ef8f1a8b1255dbb372: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260928 14:59:43.936995 13898 raft_consensus.cc:348] T 46f0e1d98e694e47ad18ea1cd309c1b9 P 99586f213ec449ef8f1a8b1255dbb372 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c34083cb0fd441ec933ad27e4602ee5e" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 38411 } } peers { permanent_uuid: "99586f213ec449ef8f1a8b1255dbb372" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 45251 } } peers { permanent_uuid: "510a02d078ff4965b5dfee7317012028" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 39831 } }
I20260928 14:59:43.937433 13898 raft_consensus.cc:374] T 46f0e1d98e694e47ad18ea1cd309c1b9 P 99586f213ec449ef8f1a8b1255dbb372 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:59:43.937551 13898 raft_consensus.cc:729] T 46f0e1d98e694e47ad18ea1cd309c1b9 P 99586f213ec449ef8f1a8b1255dbb372 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99586f213ec449ef8f1a8b1255dbb372, State: Initialized, Role: FOLLOWER
I20260928 14:59:43.938158 13898 consensus_queue.cc:260] T 46f0e1d98e694e47ad18ea1cd309c1b9 P 99586f213ec449ef8f1a8b1255dbb372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c34083cb0fd441ec933ad27e4602ee5e" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 38411 } } peers { permanent_uuid: "99586f213ec449ef8f1a8b1255dbb372" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 45251 } } peers { permanent_uuid: "510a02d078ff4965b5dfee7317012028" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 39831 } }
I20260928 14:59:43.939662 13898 ts_tablet_manager.cc:1428] T 46f0e1d98e694e47ad18ea1cd309c1b9 P 99586f213ec449ef8f1a8b1255dbb372: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
W20260928 14:59:43.993666 13609 tablet.cc:2369] T 46f0e1d98e694e47ad18ea1cd309c1b9 P 99586f213ec449ef8f1a8b1255dbb372: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:59:44.023674 13743 tablet.cc:2369] T 46f0e1d98e694e47ad18ea1cd309c1b9 P c34083cb0fd441ec933ad27e4602ee5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:59:44.038968 13877 tablet.cc:2369] T 46f0e1d98e694e47ad18ea1cd309c1b9 P 510a02d078ff4965b5dfee7317012028: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:59:44.170300 13899 raft_consensus.cc:482] T 46f0e1d98e694e47ad18ea1cd309c1b9 P c34083cb0fd441ec933ad27e4602ee5e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:59:44.170637 13899 raft_consensus.cc:504] T 46f0e1d98e694e47ad18ea1cd309c1b9 P c34083cb0fd441ec933ad27e4602ee5e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c34083cb0fd441ec933ad27e4602ee5e" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 38411 } } peers { permanent_uuid: "99586f213ec449ef8f1a8b1255dbb372" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 45251 } } peers { permanent_uuid: "510a02d078ff4965b5dfee7317012028" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 39831 } }
I20260928 14:59:44.172488 13899 leader_election.cc:288] T 46f0e1d98e694e47ad18ea1cd309c1b9 P c34083cb0fd441ec933ad27e4602ee5e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99586f213ec449ef8f1a8b1255dbb372 (127.31.40.1:45251), 510a02d078ff4965b5dfee7317012028 (127.31.40.3:39831)
I20260928 14:59:44.181198 13893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:44.173880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:58439 (local address 127.31.40.3:39831)
0928 14:59:44.174080 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:44.174085 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:44.174222 (+ 137us) server_negotiation.cc:407] Connection header received
0928 14:59:44.174472 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:44.174477 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:44.174553 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:44.174660 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:59:44.175246 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:44.176282 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:44.178732 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:44.179989 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:44.180125 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:44.180227 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:44.180803 (+ 576us) server_negotiation.cc:299] Negotiation successful
0928 14:59:44.180936 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 14:59:44.181203 13891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:44.172958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:34211 (local address 127.31.40.1:45251)
0928 14:59:44.173196 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:44.173204 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:44.173658 (+ 454us) server_negotiation.cc:407] Connection header received
0928 14:59:44.173935 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:44.173942 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:44.174028 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:44.174097 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:59:44.174925 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:44.175985 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:44.178738 (+ 2753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:44.180012 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:44.180132 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:44.180227 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:44.180803 (+ 576us) server_negotiation.cc:299] Negotiation successful
0928 14:59:44.180979 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 14:59:44.181473 13904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:44.173529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.3:39831 (local address 127.31.40.2:58439)
0928 14:59:44.174127 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:44.174145 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 14:59:44.174304 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:59:44.174692 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:59:44.174699 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:59:44.174729 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:59:44.175101 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:44.175110 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:44.176405 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:44.176414 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:44.178593 (+ 2179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:44.178601 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:44.180299 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:44.180304 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:44.180551 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:44.180653 (+ 102us) client_negotiation.cc:769] Sending connection context
0928 14:59:44.180754 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 14:59:44.180990 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"mutex_wait_us":189,"thread_start_us":311,"threads_started":1}
I20260928 14:59:44.181522 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:44.172804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:45251 (local address 127.31.40.2:34211)
0928 14:59:44.173532 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:44.173554 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 14:59:44.173718 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:59:44.174330 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:59:44.174341 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:59:44.174368 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:59:44.174758 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:44.174772 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:44.176116 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:44.176119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:44.178560 (+ 2441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:44.178578 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:44.180297 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:44.180304 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:44.180548 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:44.180659 (+ 111us) client_negotiation.cc:769] Sending connection context
0928 14:59:44.180754 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 14:59:44.180990 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"mutex_wait_us":176,"thread_start_us":369,"threads_started":1}
I20260928 14:59:44.182866 13831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46f0e1d98e694e47ad18ea1cd309c1b9" candidate_uuid: "c34083cb0fd441ec933ad27e4602ee5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "510a02d078ff4965b5dfee7317012028" is_pre_election: true
I20260928 14:59:44.182866 13563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46f0e1d98e694e47ad18ea1cd309c1b9" candidate_uuid: "c34083cb0fd441ec933ad27e4602ee5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99586f213ec449ef8f1a8b1255dbb372" is_pre_election: true
I20260928 14:59:44.183600 13831 raft_consensus.cc:2454] T 46f0e1d98e694e47ad18ea1cd309c1b9 P 510a02d078ff4965b5dfee7317012028 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c34083cb0fd441ec933ad27e4602ee5e in term 0.
I20260928 14:59:44.183600 13563 raft_consensus.cc:2454] T 46f0e1d98e694e47ad18ea1cd309c1b9 P 99586f213ec449ef8f1a8b1255dbb372 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c34083cb0fd441ec933ad27e4602ee5e in term 0.
I20260928 14:59:44.184361 13632 leader_election.cc:302] T 46f0e1d98e694e47ad18ea1cd309c1b9 P c34083cb0fd441ec933ad27e4602ee5e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 510a02d078ff4965b5dfee7317012028, c34083cb0fd441ec933ad27e4602ee5e; no voters:
I20260928 14:59:44.184777 13899 raft_consensus.cc:2781] T 46f0e1d98e694e47ad18ea1cd309c1b9 P c34083cb0fd441ec933ad27e4602ee5e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:59:44.184939 13899 raft_consensus.cc:482] T 46f0e1d98e694e47ad18ea1cd309c1b9 P c34083cb0fd441ec933ad27e4602ee5e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:59:44.185056 13899 raft_consensus.cc:3037] T 46f0e1d98e694e47ad18ea1cd309c1b9 P c34083cb0fd441ec933ad27e4602ee5e [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:59:44.187259 13899 raft_consensus.cc:504] T 46f0e1d98e694e47ad18ea1cd309c1b9 P c34083cb0fd441ec933ad27e4602ee5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c34083cb0fd441ec933ad27e4602ee5e" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 38411 } } peers { permanent_uuid: "99586f213ec449ef8f1a8b1255dbb372" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 45251 } } peers { permanent_uuid: "510a02d078ff4965b5dfee7317012028" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 39831 } }
I20260928 14:59:44.187904 13899 leader_election.cc:288] T 46f0e1d98e694e47ad18ea1cd309c1b9 P c34083cb0fd441ec933ad27e4602ee5e [CANDIDATE]: Term 1 election: Requested vote from peers 99586f213ec449ef8f1a8b1255dbb372 (127.31.40.1:45251), 510a02d078ff4965b5dfee7317012028 (127.31.40.3:39831)
I20260928 14:59:44.188386 13563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46f0e1d98e694e47ad18ea1cd309c1b9" candidate_uuid: "c34083cb0fd441ec933ad27e4602ee5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99586f213ec449ef8f1a8b1255dbb372"
I20260928 14:59:44.188470 13831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46f0e1d98e694e47ad18ea1cd309c1b9" candidate_uuid: "c34083cb0fd441ec933ad27e4602ee5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "510a02d078ff4965b5dfee7317012028"
I20260928 14:59:44.188592 13563 raft_consensus.cc:3037] T 46f0e1d98e694e47ad18ea1cd309c1b9 P 99586f213ec449ef8f1a8b1255dbb372 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:59:44.188747 13831 raft_consensus.cc:3037] T 46f0e1d98e694e47ad18ea1cd309c1b9 P 510a02d078ff4965b5dfee7317012028 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:59:44.190886 13563 raft_consensus.cc:2454] T 46f0e1d98e694e47ad18ea1cd309c1b9 P 99586f213ec449ef8f1a8b1255dbb372 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c34083cb0fd441ec933ad27e4602ee5e in term 1.
I20260928 14:59:44.191143 13831 raft_consensus.cc:2454] T 46f0e1d98e694e47ad18ea1cd309c1b9 P 510a02d078ff4965b5dfee7317012028 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c34083cb0fd441ec933ad27e4602ee5e in term 1.
I20260928 14:59:44.191373 13632 leader_election.cc:302] T 46f0e1d98e694e47ad18ea1cd309c1b9 P c34083cb0fd441ec933ad27e4602ee5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99586f213ec449ef8f1a8b1255dbb372, c34083cb0fd441ec933ad27e4602ee5e; no voters:
I20260928 14:59:44.191685 13899 raft_consensus.cc:2781] T 46f0e1d98e694e47ad18ea1cd309c1b9 P c34083cb0fd441ec933ad27e4602ee5e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:59:44.192163 13899 raft_consensus.cc:686] T 46f0e1d98e694e47ad18ea1cd309c1b9 P c34083cb0fd441ec933ad27e4602ee5e [term 1 LEADER]: Becoming Leader. State: Replica: c34083cb0fd441ec933ad27e4602ee5e, State: Running, Role: LEADER
I20260928 14:59:44.192919 13899 consensus_queue.cc:237] T 46f0e1d98e694e47ad18ea1cd309c1b9 P c34083cb0fd441ec933ad27e4602ee5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c34083cb0fd441ec933ad27e4602ee5e" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 38411 } } peers { permanent_uuid: "99586f213ec449ef8f1a8b1255dbb372" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 45251 } } peers { permanent_uuid: "510a02d078ff4965b5dfee7317012028" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 39831 } }
I20260928 14:59:44.197484 13419 catalog_manager.cc:5495] T 46f0e1d98e694e47ad18ea1cd309c1b9 P c34083cb0fd441ec933ad27e4602ee5e reported cstate change: term changed from 0 to 1, leader changed from <none> to c34083cb0fd441ec933ad27e4602ee5e (127.31.40.2). New cstate: current_term: 1 leader_uuid: "c34083cb0fd441ec933ad27e4602ee5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c34083cb0fd441ec933ad27e4602ee5e" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 38411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99586f213ec449ef8f1a8b1255dbb372" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 45251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "510a02d078ff4965b5dfee7317012028" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 39831 } health_report { overall_health: UNKNOWN } } }
I20260928 14:59:44.251120 13889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:44.245570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57632 (local address 127.31.40.2:38411)
0928 14:59:44.245812 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:44.245818 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:44.245837 (+ 19us) server_negotiation.cc:407] Connection header received
0928 14:59:44.245968 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:44.245972 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:44.246057 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:44.246221 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:59:44.246608 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:44.247316 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:44.248793 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:44.249284 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:44.249422 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:44.249926 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:59:44.250515 (+ 589us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:59:44.250689 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:44.250750 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 14:59:44.250831 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 14:59:44.259425 13563 raft_consensus.cc:1261] T 46f0e1d98e694e47ad18ea1cd309c1b9 P 99586f213ec449ef8f1a8b1255dbb372 [term 1 FOLLOWER]: Refusing update from remote peer c34083cb0fd441ec933ad27e4602ee5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:59:44.259454 13831 raft_consensus.cc:1261] T 46f0e1d98e694e47ad18ea1cd309c1b9 P 510a02d078ff4965b5dfee7317012028 [term 1 FOLLOWER]: Refusing update from remote peer c34083cb0fd441ec933ad27e4602ee5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:59:44.260478 13905 consensus_queue.cc:1035] T 46f0e1d98e694e47ad18ea1cd309c1b9 P c34083cb0fd441ec933ad27e4602ee5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "510a02d078ff4965b5dfee7317012028" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 39831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:59:44.260854 13899 consensus_queue.cc:1035] T 46f0e1d98e694e47ad18ea1cd309c1b9 P c34083cb0fd441ec933ad27e4602ee5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "99586f213ec449ef8f1a8b1255dbb372" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 45251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:59:44.271744 13908 mvcc.cc:205] Tried to move back new op lower bound from 7334328665116422144 to 7334328664858025984. Current Snapshot: MvccSnapshot[applied={T|T < 7334328665116422144}]
I20260928 14:59:44.278286 13909 mvcc.cc:205] Tried to move back new op lower bound from 7334328665116422144 to 7334328664858025984. Current Snapshot: MvccSnapshot[applied={T|T < 7334328665116422144}]
I20260928 14:59:44.311103 13419 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:59:44.311427 13419 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:59:44.315282 13419 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219: Sending DeleteTablet for 3 replicas of tablet 46f0e1d98e694e47ad18ea1cd309c1b9
I20260928 14:59:44.316702 13677 tablet_service.cc:1514] Processing DeleteTablet for tablet 46f0e1d98e694e47ad18ea1cd309c1b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:59:44 UTC) from {username='kudu', principal='kudu/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:57630
I20260928 14:59:44.316850 13811 tablet_service.cc:1514] Processing DeleteTablet for tablet 46f0e1d98e694e47ad18ea1cd309c1b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:59:44 UTC) from {username='kudu', principal='kudu/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:36978
I20260928 14:59:44.316901 13543 tablet_service.cc:1514] Processing DeleteTablet for tablet 46f0e1d98e694e47ad18ea1cd309c1b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:59:44 UTC) from {username='kudu', principal='kudu/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:41224
I20260928 14:59:44.317780 13918 tablet_replica.cc:330] T 46f0e1d98e694e47ad18ea1cd309c1b9 P 510a02d078ff4965b5dfee7317012028: stopping tablet replica
I20260928 14:59:44.318073 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 13480
I20260928 14:59:44.318429 13918 raft_consensus.cc:2229] T 46f0e1d98e694e47ad18ea1cd309c1b9 P 510a02d078ff4965b5dfee7317012028 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:59:44.319038 13918 raft_consensus.cc:2258] T 46f0e1d98e694e47ad18ea1cd309c1b9 P 510a02d078ff4965b5dfee7317012028 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:59:44.325588 13920 tablet_replica.cc:330] T 46f0e1d98e694e47ad18ea1cd309c1b9 P 99586f213ec449ef8f1a8b1255dbb372: stopping tablet replica
I20260928 14:59:44.325603 13918 ts_tablet_manager.cc:1905] T 46f0e1d98e694e47ad18ea1cd309c1b9 P 510a02d078ff4965b5dfee7317012028: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:59:44.326463 13920 raft_consensus.cc:2229] T 46f0e1d98e694e47ad18ea1cd309c1b9 P 99586f213ec449ef8f1a8b1255dbb372 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:59:44.327132 13920 raft_consensus.cc:2258] T 46f0e1d98e694e47ad18ea1cd309c1b9 P 99586f213ec449ef8f1a8b1255dbb372 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:59:44.329800 13920 ts_tablet_manager.cc:1905] T 46f0e1d98e694e47ad18ea1cd309c1b9 P 99586f213ec449ef8f1a8b1255dbb372: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:59:44.337141 13919 tablet_replica.cc:330] T 46f0e1d98e694e47ad18ea1cd309c1b9 P c34083cb0fd441ec933ad27e4602ee5e: stopping tablet replica
Sep 28 14:59:44 dist-test-slave-j968 krb5kdc[13374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.1@KRBTEST.COM
I20260928 14:59:44.338261 13919 raft_consensus.cc:2229] T 46f0e1d98e694e47ad18ea1cd309c1b9 P c34083cb0fd441ec933ad27e4602ee5e [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:59:44.338547 13920 ts_tablet_manager.cc:1918] T 46f0e1d98e694e47ad18ea1cd309c1b9 P 99586f213ec449ef8f1a8b1255dbb372: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:59:44.338800 13920 log.cc:1196] T 46f0e1d98e694e47ad18ea1cd309c1b9 P 99586f213ec449ef8f1a8b1255dbb372: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-0/wal/wals/46f0e1d98e694e47ad18ea1cd309c1b9
I20260928 14:59:44.339499 13920 ts_tablet_manager.cc:1939] T 46f0e1d98e694e47ad18ea1cd309c1b9 P 99586f213ec449ef8f1a8b1255dbb372: Deleting consensus metadata
I20260928 14:59:44.341022 13918 ts_tablet_manager.cc:1918] T 46f0e1d98e694e47ad18ea1cd309c1b9 P 510a02d078ff4965b5dfee7317012028: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:59:44.341786 13405 catalog_manager.cc:4841] TS 99586f213ec449ef8f1a8b1255dbb372 (127.31.40.1:45251): tablet 46f0e1d98e694e47ad18ea1cd309c1b9 (table test-table [id=97c1c120da104af6bd2ed3875c69979b]) successfully deleted
I20260928 14:59:44.341876 13919 raft_consensus.cc:2258] T 46f0e1d98e694e47ad18ea1cd309c1b9 P c34083cb0fd441ec933ad27e4602ee5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:59:44.342433 13918 log.cc:1196] T 46f0e1d98e694e47ad18ea1cd309c1b9 P 510a02d078ff4965b5dfee7317012028: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-2/wal/wals/46f0e1d98e694e47ad18ea1cd309c1b9
I20260928 14:59:44.343067 13918 ts_tablet_manager.cc:1939] T 46f0e1d98e694e47ad18ea1cd309c1b9 P 510a02d078ff4965b5dfee7317012028: Deleting consensus metadata
I20260928 14:59:44.344388 13919 ts_tablet_manager.cc:1905] T 46f0e1d98e694e47ad18ea1cd309c1b9 P c34083cb0fd441ec933ad27e4602ee5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:59:44.345067 13405 catalog_manager.cc:4841] TS 510a02d078ff4965b5dfee7317012028 (127.31.40.3:39831): tablet 46f0e1d98e694e47ad18ea1cd309c1b9 (table test-table [id=97c1c120da104af6bd2ed3875c69979b]) successfully deleted
I20260928 14:59:44.347271 13891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:44.319483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41240 (local address 127.31.40.1:45251)
0928 14:59:44.319727 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:44.319737 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:44.325912 (+ 6175us) server_negotiation.cc:407] Connection header received
0928 14:59:44.326242 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:44.326248 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:44.326335 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:44.326414 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:59:44.328686 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:44.329859 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:44.331909 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:44.332587 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:44.332704 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:44.339040 (+ 6336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:59:44.339063 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:59:44.339085 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:59:44.339142 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:59:44.343309 (+ 4167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:44.344404 (+ 1095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:44.344414 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:44.344418 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:44.344586 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:44.345112 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:44.345118 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:44.345121 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:44.345610 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:59:44.345918 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:44.346648 (+ 730us) server_negotiation.cc:299] Negotiation successful
0928 14:59:44.346902 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 14:59:44.347983 13603 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:59:44.352464 13919 ts_tablet_manager.cc:1918] T 46f0e1d98e694e47ad18ea1cd309c1b9 P c34083cb0fd441ec933ad27e4602ee5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:59:44.352649 13919 log.cc:1196] T 46f0e1d98e694e47ad18ea1cd309c1b9 P c34083cb0fd441ec933ad27e4602ee5e: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-1/wal/wals/46f0e1d98e694e47ad18ea1cd309c1b9
I20260928 14:59:44.353226 13919 ts_tablet_manager.cc:1939] T 46f0e1d98e694e47ad18ea1cd309c1b9 P c34083cb0fd441ec933ad27e4602ee5e: Deleting consensus metadata
I20260928 14:59:44.355221 13406 catalog_manager.cc:4841] TS c34083cb0fd441ec933ad27e4602ee5e (127.31.40.2:38411): tablet 46f0e1d98e694e47ad18ea1cd309c1b9 (table test-table [id=97c1c120da104af6bd2ed3875c69979b]) successfully deleted
I20260928 14:59:44.443879 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 13480
I20260928 14:59:44.466022 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 13614
Sep 28 14:59:44 dist-test-slave-j968 krb5kdc[13374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.2@KRBTEST.COM
I20260928 14:59:44.484133 13889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:44.466833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57634 (local address 127.31.40.2:38411)
0928 14:59:44.467022 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:44.467027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:44.467045 (+ 18us) server_negotiation.cc:407] Connection header received
0928 14:59:44.467245 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:44.467249 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:44.467312 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:44.467404 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:59:44.469183 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:44.470028 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:44.472031 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:44.472456 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:44.472555 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:44.477700 (+ 5145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:59:44.477717 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:59:44.477735 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:59:44.477782 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:59:44.481065 (+ 3283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:44.481854 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:44.481862 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:44.481865 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:44.482016 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:44.482431 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:44.482436 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:44.482444 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:44.482807 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:59:44.483177 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:44.483479 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 14:59:44.483707 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 14:59:44.484920 13737 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:59:44.600986 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 13614
I20260928 14:59:44.620395 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 13748
Sep 28 14:59:44 dist-test-slave-j968 krb5kdc[13374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.3@KRBTEST.COM
I20260928 14:59:44.635200 13893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:44.621154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36986 (local address 127.31.40.3:39831)
0928 14:59:44.621362 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:44.621366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:44.621384 (+ 18us) server_negotiation.cc:407] Connection header received
0928 14:59:44.621520 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:44.621524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:44.621587 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:44.621667 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:59:44.622951 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:44.623703 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:44.625133 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:44.625537 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:44.625625 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:44.629772 (+ 4147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:59:44.629787 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:59:44.629804 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:59:44.629843 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:59:44.632528 (+ 2685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:44.633182 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:44.633190 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:44.633192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:44.633375 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:44.633696 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:44.633699 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:44.633701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:44.634071 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:59:44.634310 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:44.634666 (+ 356us) server_negotiation.cc:299] Negotiation successful
0928 14:59:44.634826 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260928 14:59:44.635953 13871 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:59:44.727572 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 13748
I20260928 14:59:44.745932 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 13387
I20260928 14:59:44.747112 13449 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:59:44.860252 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 13387
I20260928 14:59:44.884440 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:43875
--webserver_interface=127.31.40.62
--webserver_port=36845
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:39673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.40.62:43875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:59:45.227763 13926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:59:45.228158 13926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:59:45.228331 13926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:59:45.235774 13926 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:59:45.235966 13926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:59:45.236080 13926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:59:45.236186 13926 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:59:45.236267 13926 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:59:45.246569 13926 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:39673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.31.40.62:43875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:43875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.40.62
--webserver_port=36845
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:59:45.248070 13926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:59:45.249440 13926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:59:45.274019 13933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:59:45.274035 13935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:59:45.274385 13926 server_base.cc:1034] running on GCE node
W20260928 14:59:45.274103 13932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:59:45.275635 13926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:59:45.277452 13926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:59:45.278726 13926 hybrid_clock.cc:648] HybridClock initialized: now 1790607585278683 us; error 57 us; skew 500 ppm
I20260928 14:59:45.279227 13926 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:59:45 dist-test-slave-j968 krb5kdc[13374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607585, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:59:45.288491 13926 init.cc:377] Logged in from keytab as kudu/127.31.40.62@KRBTEST.COM (short username kudu)
I20260928 14:59:45.290508 13926 webserver.cc:460] Webserver started at http://127.31.40.62:36845/ using document root <none> and password file <none>
I20260928 14:59:45.291348 13926 fs_manager.cc:359] Metadata directory not provided
I20260928 14:59:45.291479 13926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:59:45.298286 13926 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260928 14:59:45.302058 13942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:59:45.303763 13926 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 14:59:45.304031 13926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/master-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/master-0/wal
uuid: "40864fbc798f4c4b8033a199f800a219"
format_stamp: "Formatted at 2026-09-28 14:59:42 on dist-test-slave-j968"
server_key: "463f55961d213e59e4bd0704c14b73f7"
server_key_iv: "527ed252030efd282a04fd5f1b04591d"
server_key_version: "encryptionkey@0"
I20260928 14:59:45.305166 13926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:59:45.340037 13926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:59:45.346138 13926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:59:45.346470 13926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:59:45.371730 13926 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.62:43875
I20260928 14:59:45.371762 13993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.62:43875 every 8 connection(s)
I20260928 14:59:45.373953 13926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
I20260928 14:59:45.380757 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 13926
I20260928 14:59:45.381976 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.1:45251
--local_ip_for_outbound_sockets=127.31.40.1
--tserver_master_addrs=127.31.40.62:43875
--webserver_port=36631
--webserver_interface=127.31.40.1
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab
--principal=kudu/127.31.40.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:39673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:59:45.383994 13994 sys_catalog.cc:263] Verifying existing consensus state
I20260928 14:59:45.387547 13994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219: Bootstrap starting.
I20260928 14:59:45.409893 13994 log.cc:824] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219: Log is configured to *not* fsync() on all Append() calls
I20260928 14:59:45.422823 13994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:59:45.423386 13994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219: Bootstrap complete.
I20260928 14:59:45.433315 13994 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40864fbc798f4c4b8033a199f800a219" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 43875 } }
I20260928 14:59:45.434223 13994 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40864fbc798f4c4b8033a199f800a219, State: Initialized, Role: FOLLOWER
I20260928 14:59:45.434819 13994 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40864fbc798f4c4b8033a199f800a219" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 43875 } }
I20260928 14:59:45.435191 13994 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:59:45.435393 13994 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:59:45.435647 13994 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219 [term 1 FOLLOWER]: Advancing to term 2
I20260928 14:59:45.441278 13994 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40864fbc798f4c4b8033a199f800a219" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 43875 } }
I20260928 14:59:45.442001 13994 leader_election.cc:302] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40864fbc798f4c4b8033a199f800a219; no voters:
I20260928 14:59:45.442728 13994 leader_election.cc:288] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 14:59:45.443009 13998 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 14:59:45.446326 13998 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219 [term 2 LEADER]: Becoming Leader. State: Replica: 40864fbc798f4c4b8033a199f800a219, State: Running, Role: LEADER
I20260928 14:59:45.446985 13998 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40864fbc798f4c4b8033a199f800a219" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 43875 } }
I20260928 14:59:45.448892 13994 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:59:45.451730 13999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40864fbc798f4c4b8033a199f800a219. Latest consensus state: current_term: 2 leader_uuid: "40864fbc798f4c4b8033a199f800a219" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40864fbc798f4c4b8033a199f800a219" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 43875 } } }
I20260928 14:59:45.452283 13999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219 [sys.catalog]: This master's current role is: LEADER
I20260928 14:59:45.452145 14000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "40864fbc798f4c4b8033a199f800a219" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40864fbc798f4c4b8033a199f800a219" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 43875 } } }
I20260928 14:59:45.452814 14000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219 [sys.catalog]: This master's current role is: LEADER
I20260928 14:59:45.460527 14005 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:59:45.472769 14005 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:59:45.474752 14005 catalog_manager.cc:1259] Loaded cluster ID: dd0d8e389e114f4fbaf6eb4039ae60a7
I20260928 14:59:45.474946 14005 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:59:45.478471 14005 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 14:59:45.496727 14005 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 14:59:45.506973 14005 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:59:45.509133 14005 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219: Loaded TSK: 0
I20260928 14:59:45.511497 14005 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 14:59:45.775013 13996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:59:45.775384 13996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:59:45.775527 13996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:59:45.783432 13996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:59:45.783618 13996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:59:45.783808 13996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.1
I20260928 14:59:45.794200 13996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:39673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab
--principal=kudu/127.31.40.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.1:45251
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.40.1
--webserver_port=36631
--tserver_master_addrs=127.31.40.62:43875
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.1
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:59:45.795706 13996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:59:45.797233 13996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:59:45.813511 14024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:59:45.813583 14022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:59:45.813453 14021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:59:45.815191 13996 server_base.cc:1034] running on GCE node
I20260928 14:59:45.816074 13996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:59:45.817338 13996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:59:45.818578 13996 hybrid_clock.cc:648] HybridClock initialized: now 1790607585818521 us; error 46 us; skew 500 ppm
I20260928 14:59:45.818998 13996 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:59:45 dist-test-slave-j968 krb5kdc[13374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607585, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:59:45.826939 13996 init.cc:377] Logged in from keytab as kudu/127.31.40.1@KRBTEST.COM (short username kudu)
I20260928 14:59:45.828812 13996 webserver.cc:460] Webserver started at http://127.31.40.1:36631/ using document root <none> and password file <none>
I20260928 14:59:45.829610 13996 fs_manager.cc:359] Metadata directory not provided
I20260928 14:59:45.829771 13996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:59:45.835978 13996 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 14:59:45.839147 14031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:59:45.840880 13996 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:59:45.841109 13996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-0/wal
uuid: "99586f213ec449ef8f1a8b1255dbb372"
format_stamp: "Formatted at 2026-09-28 14:59:42 on dist-test-slave-j968"
server_key: "b0b4b056fd8add989e3eb7e289374c91"
server_key_iv: "a9788ed67f36beaa5d647573f7b9802c"
server_key_version: "encryptionkey@0"
I20260928 14:59:45.842283 13996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:59:45.872537 13996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:59:45.877496 13996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:59:45.877818 13996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:59:45.878942 13996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:59:45.880424 13996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:59:45.880515 13996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:59:45.880590 13996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:59:45.880645 13996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:59:45.916163 13996 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.1:45251
I20260928 14:59:45.916214 14143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.1:45251 every 8 connection(s)
I20260928 14:59:45.917768 13996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
I20260928 14:59:45.919369 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 13996
I20260928 14:59:45.920233 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.2:38411
--local_ip_for_outbound_sockets=127.31.40.2
--tserver_master_addrs=127.31.40.62:43875
--webserver_port=39369
--webserver_interface=127.31.40.2
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab
--principal=kudu/127.31.40.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:39673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 14:59:45 dist-test-slave-j968 krb5kdc[13374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607585, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.1@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:59:45.944226 14146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:45.921518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:60667 (local address 127.31.40.62:43875)
0928 14:59:45.922202 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:45.922217 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:45.923017 (+ 800us) server_negotiation.cc:407] Connection header received
0928 14:59:45.925521 (+ 2504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:45.925535 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:45.925934 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:45.926343 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:59:45.928143 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:45.929490 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:45.931170 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:45.931840 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:45.932127 (+ 287us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:45.935975 (+ 3848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:59:45.935987 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:59:45.936005 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:59:45.936055 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:59:45.939144 (+ 3089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:45.940064 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:45.940075 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:45.940088 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:45.940266 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:45.940756 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:45.940761 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:45.940764 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:45.941297 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:59:45.941527 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:45.942395 (+ 868us) server_negotiation.cc:299] Negotiation successful
0928 14:59:45.942821 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":325,"threads_started":1}
I20260928 14:59:45.944226 14147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:45.921972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.1:60667)
0928 14:59:45.922808 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:45.922857 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 14:59:45.924874 (+ 2017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:59:45.926542 (+ 1668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:59:45.926553 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:59:45.927229 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:59:45.927828 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:45.927847 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:45.929692 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:45.929703 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:45.930992 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:45.931018 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:45.932169 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:45.932176 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:45.932473 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:45.933220 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:59:45.933245 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:59:45.935585 (+ 2340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:59:45.939326 (+ 3741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:59:45.939336 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:59:45.939354 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:59:45.939828 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:59:45.940408 (+ 580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:59:45.940413 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:59:45.940417 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:59:45.940623 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:59:45.941478 (+ 855us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:59:45.941487 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:59:45.941874 (+ 387us) client_negotiation.cc:769] Sending connection context
0928 14:59:45.942160 (+ 286us) client_negotiation.cc:240] Negotiation successful
0928 14:59:45.942790 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":340,"threads_started":1}
I20260928 14:59:45.947535 14144 heartbeater.cc:347] Connected to a master server at 127.31.40.62:43875
I20260928 14:59:45.947898 14144 heartbeater.cc:464] Registering TS with master...
I20260928 14:59:45.948683 14144 heartbeater.cc:511] Master 127.31.40.62:43875 requested a full tablet report, sending...
I20260928 14:59:45.951081 13959 ts_manager.cc:194] Registered new tserver with Master: 99586f213ec449ef8f1a8b1255dbb372 (127.31.40.1:45251)
I20260928 14:59:45.952689 13959 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.1@KRBTEST.COM'} at 127.31.40.1:60667
W20260928 14:59:46.275446 14148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:59:46.275784 14148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:59:46.275992 14148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:59:46.283499 14148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:59:46.283691 14148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:59:46.283908 14148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.2
I20260928 14:59:46.294668 14148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:39673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab
--principal=kudu/127.31.40.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.2:38411
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.40.2
--webserver_port=39369
--tserver_master_addrs=127.31.40.62:43875
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.2
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:59:46.296131 14148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:59:46.297580 14148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:59:46.311885 14153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:59:46.311858 14156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:59:46.311944 14154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:59:46.312654 14148 server_base.cc:1034] running on GCE node
I20260928 14:59:46.313392 14148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:59:46.314465 14148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:59:46.315722 14148 hybrid_clock.cc:648] HybridClock initialized: now 1790607586315653 us; error 63 us; skew 500 ppm
I20260928 14:59:46.316146 14148 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:59:46 dist-test-slave-j968 krb5kdc[13374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607586, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:59:46.319557 14148 init.cc:377] Logged in from keytab as kudu/127.31.40.2@KRBTEST.COM (short username kudu)
I20260928 14:59:46.321231 14148 webserver.cc:460] Webserver started at http://127.31.40.2:39369/ using document root <none> and password file <none>
I20260928 14:59:46.322049 14148 fs_manager.cc:359] Metadata directory not provided
I20260928 14:59:46.322232 14148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:59:46.328042 14148 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 14:59:46.330997 14163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:59:46.332563 14148 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:59:46.332752 14148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-1/wal
uuid: "c34083cb0fd441ec933ad27e4602ee5e"
format_stamp: "Formatted at 2026-09-28 14:59:43 on dist-test-slave-j968"
server_key: "21ed01889dc007b0666c606660d5b18e"
server_key_iv: "7d60ad8a36b4fedaac4945143323055b"
server_key_version: "encryptionkey@0"
I20260928 14:59:46.333863 14148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:59:46.354548 14148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:59:46.359094 14148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:59:46.359388 14148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:59:46.360455 14148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:59:46.361824 14148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:59:46.361959 14148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:59:46.362112 14148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:59:46.362205 14148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:59:46.397862 14148 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.2:38411
I20260928 14:59:46.397922 14275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.2:38411 every 8 connection(s)
I20260928 14:59:46.399375 14148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
I20260928 14:59:46.408022 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 14148
I20260928 14:59:46.408681 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.3:39831
--local_ip_for_outbound_sockets=127.31.40.3
--tserver_master_addrs=127.31.40.62:43875
--webserver_port=43059
--webserver_interface=127.31.40.3
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab
--principal=kudu/127.31.40.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:39673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 14:59:46 dist-test-slave-j968 krb5kdc[13374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607586, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.2@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:59:46.421358 14146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:46.402130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:52581 (local address 127.31.40.62:43875)
0928 14:59:46.402346 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:46.402350 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:46.403574 (+ 1224us) server_negotiation.cc:407] Connection header received
0928 14:59:46.405596 (+ 2022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:46.405602 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:46.405751 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:46.405881 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:59:46.407517 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:46.408383 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:46.410322 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:46.410845 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:46.410991 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:46.414722 (+ 3731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:59:46.414729 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:59:46.414733 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:59:46.414778 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:59:46.417987 (+ 3209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:46.418771 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:46.418779 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:46.418783 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:46.418926 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:46.419422 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:46.419427 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:46.419430 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:46.419714 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:59:46.419882 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:46.420679 (+ 797us) server_negotiation.cc:299] Negotiation successful
0928 14:59:46.420947 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 14:59:46.422489 14278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:46.402560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.2:52581)
0928 14:59:46.403386 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:46.403428 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 14:59:46.405273 (+ 1845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:59:46.406153 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:59:46.406162 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:59:46.406696 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:59:46.407322 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:46.407340 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:46.408565 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:46.408569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:46.410143 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:46.410157 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:46.411102 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:46.411110 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:46.411368 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:46.412124 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:59:46.412154 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:59:46.414483 (+ 2329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:59:46.418142 (+ 3659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:59:46.418152 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:59:46.418171 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:59:46.418633 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:59:46.419066 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:59:46.419071 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:59:46.419074 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:59:46.419286 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:59:46.419885 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:59:46.419895 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:59:46.420268 (+ 373us) client_negotiation.cc:769] Sending connection context
0928 14:59:46.420570 (+ 302us) client_negotiation.cc:240] Negotiation successful
0928 14:59:46.421209 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":353,"threads_started":1}
I20260928 14:59:46.424296 14276 heartbeater.cc:347] Connected to a master server at 127.31.40.62:43875
I20260928 14:59:46.424655 14276 heartbeater.cc:464] Registering TS with master...
I20260928 14:59:46.425726 14276 heartbeater.cc:511] Master 127.31.40.62:43875 requested a full tablet report, sending...
I20260928 14:59:46.427639 13959 ts_manager.cc:194] Registered new tserver with Master: c34083cb0fd441ec933ad27e4602ee5e (127.31.40.2:38411)
I20260928 14:59:46.429024 13959 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.2@KRBTEST.COM'} at 127.31.40.2:52581
W20260928 14:59:46.759370 14279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:59:46.759660 14279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:59:46.759756 14279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:59:46.767481 14279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:59:46.767604 14279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:59:46.767721 14279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.3
I20260928 14:59:46.778108 14279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:39673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab
--principal=kudu/127.31.40.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.3:39831
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.40.3
--webserver_port=43059
--tserver_master_addrs=127.31.40.62:43875
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.3
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:59:46.779408 14279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:59:46.780833 14279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:59:46.795104 14285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:59:46.795202 14287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:59:46.795252 14284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:59:46.795586 14279 server_base.cc:1034] running on GCE node
I20260928 14:59:46.796479 14279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:59:46.797559 14279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:59:46.798825 14279 hybrid_clock.cc:648] HybridClock initialized: now 1790607586798751 us; error 62 us; skew 500 ppm
I20260928 14:59:46.799203 14279 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:59:46 dist-test-slave-j968 krb5kdc[13374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607586, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:59:46.802915 14279 init.cc:377] Logged in from keytab as kudu/127.31.40.3@KRBTEST.COM (short username kudu)
I20260928 14:59:46.804512 14279 webserver.cc:460] Webserver started at http://127.31.40.3:43059/ using document root <none> and password file <none>
I20260928 14:59:46.805235 14279 fs_manager.cc:359] Metadata directory not provided
I20260928 14:59:46.805370 14279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:59:46.811510 14279 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 14:59:46.814551 14294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:59:46.816059 14279 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:59:46.816295 14279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-2/wal
uuid: "510a02d078ff4965b5dfee7317012028"
format_stamp: "Formatted at 2026-09-28 14:59:43 on dist-test-slave-j968"
server_key: "67992f9ae5900758844c16c37b5a00d6"
server_key_iv: "194cf94a9a7262c84a9b9ae52218bd35"
server_key_version: "encryptionkey@0"
I20260928 14:59:46.817309 14279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:59:46.859688 14279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:59:46.864553 14279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:59:46.864789 14279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:59:46.865801 14279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:59:46.867121 14279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:59:46.867219 14279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:59:46.867296 14279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:59:46.867352 14279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:59:46.901473 14279 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.3:39831
I20260928 14:59:46.901590 14406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.3:39831 every 8 connection(s)
I20260928 14:59:46.903038 14279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
I20260928 14:59:46.904891 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 14279
Sep 28 14:59:46 dist-test-slave-j968 krb5kdc[13374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607586, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.3@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 14:59:46.924211 14410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:46.905944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55258 (local address 127.31.40.62:43875)
0928 14:59:46.906460 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:46.906465 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:46.906891 (+ 426us) server_negotiation.cc:407] Connection header received
0928 14:59:46.907103 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:46.907108 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:46.907192 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:46.907314 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:59:46.909553 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:46.910541 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:46.912594 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:46.913077 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:46.913220 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:46.916669 (+ 3449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:59:46.916673 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:59:46.916676 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:59:46.916704 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:59:46.919829 (+ 3125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:46.920739 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:46.920745 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:46.920749 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:46.920923 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:46.921927 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:46.921934 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:46.921937 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:46.922377 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:59:46.922674 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:46.923075 (+ 401us) server_negotiation.cc:299] Negotiation successful
0928 14:59:46.923696 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":90,"server-negotiator.queue_time_us":418,"spinlock_wait_cycles":20992,"thread_start_us":240,"threads_started":1}
I20260928 14:59:46.927619 14146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:46.905944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:56333 (local address 127.31.40.62:43875)
0928 14:59:46.906112 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:46.906117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:46.907681 (+ 1564us) server_negotiation.cc:407] Connection header received
0928 14:59:46.910237 (+ 2556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:46.910243 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:46.910334 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:46.910468 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:59:46.912534 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:46.913291 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:46.915472 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:46.916204 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:46.916345 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:46.920814 (+ 4469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:59:46.920822 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:59:46.920830 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:59:46.920877 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:59:46.924509 (+ 3632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:46.925288 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:46.925294 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:46.925297 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:46.925429 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:46.925888 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:46.925893 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:46.925896 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:46.926159 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:59:46.926319 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:46.927032 (+ 713us) server_negotiation.cc:299] Negotiation successful
0928 14:59:46.927248 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":37}
I20260928 14:59:46.928709 14411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:46.906605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.3:56333)
0928 14:59:46.907510 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:46.907571 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 14:59:46.909954 (+ 2383us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:59:46.910769 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:59:46.910783 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:59:46.911526 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:59:46.912361 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:46.912385 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:46.913502 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:46.913512 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:46.915302 (+ 1790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:46.915318 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:46.916457 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:46.916466 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:46.916771 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:46.917527 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:59:46.917565 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:59:46.920519 (+ 2954us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:59:46.924662 (+ 4143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:59:46.924671 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:59:46.924690 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:59:46.925149 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:59:46.925550 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:59:46.925556 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:59:46.925560 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:59:46.925760 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:59:46.926307 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:59:46.926316 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:59:46.926662 (+ 346us) client_negotiation.cc:769] Sending connection context
0928 14:59:46.926939 (+ 277us) client_negotiation.cc:240] Negotiation successful
0928 14:59:46.927509 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":353,"threads_started":1}
I20260928 14:59:46.929997 14407 heartbeater.cc:347] Connected to a master server at 127.31.40.62:43875
I20260928 14:59:46.930325 14407 heartbeater.cc:464] Registering TS with master...
I20260928 14:59:46.931100 14407 heartbeater.cc:511] Master 127.31.40.62:43875 requested a full tablet report, sending...
I20260928 14:59:46.932694 13959 ts_manager.cc:194] Registered new tserver with Master: 510a02d078ff4965b5dfee7317012028 (127.31.40.3:39831)
I20260928 14:59:46.933594 13959 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.3@KRBTEST.COM'} at 127.31.40.3:56333
I20260928 14:59:46.939373 31904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:59:46.946291 14146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:46.940375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55266 (local address 127.31.40.62:43875)
0928 14:59:46.940576 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:46.940580 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:46.941095 (+ 515us) server_negotiation.cc:407] Connection header received
0928 14:59:46.941266 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:46.941270 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:46.941416 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:46.941542 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:59:46.941970 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:46.942723 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:46.944280 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:46.944705 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:46.944803 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:46.945251 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:59:46.945746 (+ 495us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:59:46.945852 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:46.946014 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 14:59:46.946102 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260928 14:59:46.947214 13959 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:55266:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:59:46.954427 13959 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:59:46.955932 14144 heartbeater.cc:503] Master 127.31.40.62:43875 was elected leader, sending a full tablet report...
I20260928 14:59:46.987207 14422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:46.969670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57648 (local address 127.31.40.2:38411)
0928 14:59:46.977599 (+ 7929us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:46.977613 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:46.979721 (+ 2108us) server_negotiation.cc:407] Connection header received
0928 14:59:46.979834 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:46.979844 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:46.980114 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:46.980298 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:59:46.980961 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:46.982506 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:46.984703 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:46.986004 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:46.986156 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:46.986383 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:46.986678 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 14:59:46.986845 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7672,"thread_start_us":8170,"threads_started":1}
I20260928 14:59:46.987263 14419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:46.968926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.2:38411 (local address 127.0.0.1:57648)
0928 14:59:46.979443 (+ 10517us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:46.979460 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:59:46.979581 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:59:46.980296 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:59:46.980300 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:59:46.980322 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:59:46.980746 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:46.980760 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:46.982646 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:46.982655 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:46.984472 (+ 1817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:46.984484 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:46.986216 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:46.986222 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:46.986423 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:46.986481 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 14:59:46.986636 (+ 155us) client_negotiation.cc:240] Negotiation successful
0928 14:59:46.986832 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10391,"mutex_wait_us":27,"thread_start_us":7324,"threads_started":1}
I20260928 14:59:46.987923 14417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:46.967149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.3:39831 (local address 127.0.0.1:36988)
0928 14:59:46.968077 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:46.968127 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 14:59:46.968293 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:59:46.979471 (+ 11178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:59:46.979483 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:59:46.979521 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:59:46.980046 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:46.980060 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:46.982424 (+ 2364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:46.982430 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:46.984771 (+ 2341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:46.984784 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:46.986921 (+ 2137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:46.986929 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:46.987305 (+ 376us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:46.987338 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 14:59:46.987408 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 14:59:46.987512 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"mutex_wait_us":114,"thread_start_us":488,"threads_started":1}
I20260928 14:59:46.988186 14420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:46.968991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41244 (local address 127.31.40.1:45251)
0928 14:59:46.979009 (+ 10018us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:46.979017 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:46.979041 (+ 24us) server_negotiation.cc:407] Connection header received
0928 14:59:46.979132 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:46.979139 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:46.979359 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:46.979553 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:59:46.980469 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:46.981867 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:46.984843 (+ 2976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:46.986342 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:46.986474 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:46.986708 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:46.987636 (+ 928us) server_negotiation.cc:299] Negotiation successful
0928 14:59:46.987806 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9833,"thread_start_us":9698,"threads_started":1}
I20260928 14:59:46.988873 14421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:46.968560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:45251 (local address 127.0.0.1:41244)
0928 14:59:46.976705 (+ 8145us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:46.976734 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 14:59:46.976926 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:59:46.979718 (+ 2792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:59:46.979724 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:59:46.979752 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:59:46.980246 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:46.980259 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:46.982017 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:46.982023 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:46.984644 (+ 2621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:46.984661 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:46.986935 (+ 2274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:46.986941 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:46.987152 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:46.987294 (+ 142us) client_negotiation.cc:769] Sending connection context
0928 14:59:46.987374 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 14:59:46.987508 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7913,"mutex_wait_us":171,"thread_start_us":10634,"threads_started":1}
I20260928 14:59:46.989634 14418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:46.967230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36988 (local address 127.31.40.3:39831)
0928 14:59:46.978669 (+ 11439us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:46.978680 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:46.978722 (+ 42us) server_negotiation.cc:407] Connection header received
0928 14:59:46.978838 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:46.978846 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:46.979146 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:46.979360 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:59:46.980592 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:46.982277 (+ 1685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:46.984945 (+ 2668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:46.986342 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:46.986475 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:46.986707 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:46.989096 (+ 2389us) server_negotiation.cc:299] Negotiation successful
0928 14:59:46.989254 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11253,"thread_start_us":11092,"threads_started":1}
I20260928 14:59:46.991967 14211 tablet_service.cc:1467] Processing CreateTablet for tablet 40875a7e94c840fdbfa8e5dc020db5b3 (DEFAULT_TABLE table=test-table [id=76201495cba041c093a151098b060349]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:59:46.992321 14079 tablet_service.cc:1467] Processing CreateTablet for tablet 40875a7e94c840fdbfa8e5dc020db5b3 (DEFAULT_TABLE table=test-table [id=76201495cba041c093a151098b060349]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:59:46.993556 14079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40875a7e94c840fdbfa8e5dc020db5b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:59:46.993556 14211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40875a7e94c840fdbfa8e5dc020db5b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:59:46.993306 14342 tablet_service.cc:1467] Processing CreateTablet for tablet 40875a7e94c840fdbfa8e5dc020db5b3 (DEFAULT_TABLE table=test-table [id=76201495cba041c093a151098b060349]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:59:46.994637 14342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40875a7e94c840fdbfa8e5dc020db5b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:59:47.005816 14423 tablet_bootstrap.cc:492] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372: Bootstrap starting.
I20260928 14:59:47.009162 14424 tablet_bootstrap.cc:492] T 40875a7e94c840fdbfa8e5dc020db5b3 P c34083cb0fd441ec933ad27e4602ee5e: Bootstrap starting.
I20260928 14:59:47.009492 14423 tablet_bootstrap.cc:654] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372: Neither blocks nor log segments found. Creating new log.
I20260928 14:59:47.010900 14423 log.cc:824] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372: Log is configured to *not* fsync() on all Append() calls
I20260928 14:59:47.011183 14425 tablet_bootstrap.cc:492] T 40875a7e94c840fdbfa8e5dc020db5b3 P 510a02d078ff4965b5dfee7317012028: Bootstrap starting.
I20260928 14:59:47.013399 14424 tablet_bootstrap.cc:654] T 40875a7e94c840fdbfa8e5dc020db5b3 P c34083cb0fd441ec933ad27e4602ee5e: Neither blocks nor log segments found. Creating new log.
I20260928 14:59:47.014402 14425 tablet_bootstrap.cc:654] T 40875a7e94c840fdbfa8e5dc020db5b3 P 510a02d078ff4965b5dfee7317012028: Neither blocks nor log segments found. Creating new log.
I20260928 14:59:47.014985 14424 log.cc:824] T 40875a7e94c840fdbfa8e5dc020db5b3 P c34083cb0fd441ec933ad27e4602ee5e: Log is configured to *not* fsync() on all Append() calls
I20260928 14:59:47.015594 14425 log.cc:824] T 40875a7e94c840fdbfa8e5dc020db5b3 P 510a02d078ff4965b5dfee7317012028: Log is configured to *not* fsync() on all Append() calls
I20260928 14:59:47.018352 14425 tablet_bootstrap.cc:492] T 40875a7e94c840fdbfa8e5dc020db5b3 P 510a02d078ff4965b5dfee7317012028: No bootstrap required, opened a new log
I20260928 14:59:47.018505 14424 tablet_bootstrap.cc:492] T 40875a7e94c840fdbfa8e5dc020db5b3 P c34083cb0fd441ec933ad27e4602ee5e: No bootstrap required, opened a new log
I20260928 14:59:47.018522 14423 tablet_bootstrap.cc:492] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372: No bootstrap required, opened a new log
I20260928 14:59:47.018685 14425 ts_tablet_manager.cc:1397] T 40875a7e94c840fdbfa8e5dc020db5b3 P 510a02d078ff4965b5dfee7317012028: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260928 14:59:47.018782 14424 ts_tablet_manager.cc:1397] T 40875a7e94c840fdbfa8e5dc020db5b3 P c34083cb0fd441ec933ad27e4602ee5e: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260928 14:59:47.018782 14423 ts_tablet_manager.cc:1397] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.010s
I20260928 14:59:47.026441 14425 raft_consensus.cc:348] T 40875a7e94c840fdbfa8e5dc020db5b3 P 510a02d078ff4965b5dfee7317012028 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "510a02d078ff4965b5dfee7317012028" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 39831 } } peers { permanent_uuid: "99586f213ec449ef8f1a8b1255dbb372" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 45251 } } peers { permanent_uuid: "c34083cb0fd441ec933ad27e4602ee5e" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 38411 } }
I20260928 14:59:47.026839 14425 raft_consensus.cc:374] T 40875a7e94c840fdbfa8e5dc020db5b3 P 510a02d078ff4965b5dfee7317012028 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:59:47.027035 14425 raft_consensus.cc:729] T 40875a7e94c840fdbfa8e5dc020db5b3 P 510a02d078ff4965b5dfee7317012028 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 510a02d078ff4965b5dfee7317012028, State: Initialized, Role: FOLLOWER
I20260928 14:59:47.027707 14425 consensus_queue.cc:260] T 40875a7e94c840fdbfa8e5dc020db5b3 P 510a02d078ff4965b5dfee7317012028 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "510a02d078ff4965b5dfee7317012028" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 39831 } } peers { permanent_uuid: "99586f213ec449ef8f1a8b1255dbb372" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 45251 } } peers { permanent_uuid: "c34083cb0fd441ec933ad27e4602ee5e" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 38411 } }
I20260928 14:59:47.028930 14407 heartbeater.cc:503] Master 127.31.40.62:43875 was elected leader, sending a full tablet report...
I20260928 14:59:47.029605 14425 ts_tablet_manager.cc:1428] T 40875a7e94c840fdbfa8e5dc020db5b3 P 510a02d078ff4965b5dfee7317012028: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260928 14:59:47.029654 14423 raft_consensus.cc:348] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "510a02d078ff4965b5dfee7317012028" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 39831 } } peers { permanent_uuid: "99586f213ec449ef8f1a8b1255dbb372" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 45251 } } peers { permanent_uuid: "c34083cb0fd441ec933ad27e4602ee5e" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 38411 } }
I20260928 14:59:47.029605 14424 raft_consensus.cc:348] T 40875a7e94c840fdbfa8e5dc020db5b3 P c34083cb0fd441ec933ad27e4602ee5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "510a02d078ff4965b5dfee7317012028" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 39831 } } peers { permanent_uuid: "99586f213ec449ef8f1a8b1255dbb372" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 45251 } } peers { permanent_uuid: "c34083cb0fd441ec933ad27e4602ee5e" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 38411 } }
I20260928 14:59:47.030058 14423 raft_consensus.cc:374] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:59:47.030058 14424 raft_consensus.cc:374] T 40875a7e94c840fdbfa8e5dc020db5b3 P c34083cb0fd441ec933ad27e4602ee5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:59:47.030154 14423 raft_consensus.cc:729] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99586f213ec449ef8f1a8b1255dbb372, State: Initialized, Role: FOLLOWER
I20260928 14:59:47.030154 14424 raft_consensus.cc:729] T 40875a7e94c840fdbfa8e5dc020db5b3 P c34083cb0fd441ec933ad27e4602ee5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c34083cb0fd441ec933ad27e4602ee5e, State: Initialized, Role: FOLLOWER
I20260928 14:59:47.030838 14424 consensus_queue.cc:260] T 40875a7e94c840fdbfa8e5dc020db5b3 P c34083cb0fd441ec933ad27e4602ee5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "510a02d078ff4965b5dfee7317012028" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 39831 } } peers { permanent_uuid: "99586f213ec449ef8f1a8b1255dbb372" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 45251 } } peers { permanent_uuid: "c34083cb0fd441ec933ad27e4602ee5e" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 38411 } }
I20260928 14:59:47.030840 14423 consensus_queue.cc:260] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "510a02d078ff4965b5dfee7317012028" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 39831 } } peers { permanent_uuid: "99586f213ec449ef8f1a8b1255dbb372" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 45251 } } peers { permanent_uuid: "c34083cb0fd441ec933ad27e4602ee5e" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 38411 } }
I20260928 14:59:47.032652 14424 ts_tablet_manager.cc:1428] T 40875a7e94c840fdbfa8e5dc020db5b3 P c34083cb0fd441ec933ad27e4602ee5e: Time spent starting tablet: real 0.014s user 0.009s sys 0.005s
I20260928 14:59:47.033120 14423 ts_tablet_manager.cc:1428] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372: Time spent starting tablet: real 0.014s user 0.014s sys 0.002s
I20260928 14:59:47.034404 14276 heartbeater.cc:503] Master 127.31.40.62:43875 was elected leader, sending a full tablet report...
I20260928 14:59:47.107239 14429 raft_consensus.cc:482] T 40875a7e94c840fdbfa8e5dc020db5b3 P 510a02d078ff4965b5dfee7317012028 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:59:47.107653 14429 raft_consensus.cc:504] T 40875a7e94c840fdbfa8e5dc020db5b3 P 510a02d078ff4965b5dfee7317012028 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "510a02d078ff4965b5dfee7317012028" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 39831 } } peers { permanent_uuid: "99586f213ec449ef8f1a8b1255dbb372" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 45251 } } peers { permanent_uuid: "c34083cb0fd441ec933ad27e4602ee5e" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 38411 } }
I20260928 14:59:47.110155 14429 leader_election.cc:288] T 40875a7e94c840fdbfa8e5dc020db5b3 P 510a02d078ff4965b5dfee7317012028 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99586f213ec449ef8f1a8b1255dbb372 (127.31.40.1:45251), c34083cb0fd441ec933ad27e4602ee5e (127.31.40.2:38411)
I20260928 14:59:47.118359 14420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:47.111071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:55839 (local address 127.31.40.1:45251)
0928 14:59:47.111924 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:47.111931 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:47.111954 (+ 23us) server_negotiation.cc:407] Connection header received
0928 14:59:47.112034 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:47.112040 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:47.112126 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:47.112226 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:59:47.112805 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:47.113972 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:47.116559 (+ 2587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:47.117435 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:47.117591 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:47.117696 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:47.117963 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 14:59:47.118096 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653}
I20260928 14:59:47.118366 14411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:47.110480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:45251 (local address 127.31.40.3:55839)
0928 14:59:47.111352 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:47.111376 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 14:59:47.111525 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:59:47.112337 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:59:47.112346 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:59:47.112385 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:59:47.112661 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:47.112674 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:47.114131 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:47.114140 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:47.116400 (+ 2260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:47.116412 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:47.117602 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:47.117608 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:47.117796 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:47.117820 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 14:59:47.117935 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 14:59:47.118067 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"mutex_wait_us":39}
I20260928 14:59:47.119061 14432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:47.110837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.2:38411 (local address 127.31.40.3:50723)
0928 14:59:47.112917 (+ 2080us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:47.112936 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 14:59:47.113069 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:59:47.113502 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:59:47.113507 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:59:47.113547 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:59:47.113884 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:47.113894 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:47.115310 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:47.115316 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:47.117175 (+ 1859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:47.117188 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:47.118302 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:47.118308 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:47.118521 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:47.118568 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 14:59:47.118657 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 14:59:47.118777 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1924,"mutex_wait_us":1,"thread_start_us":1675,"threads_started":1}
I20260928 14:59:47.119096 14422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:47.110653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:50723 (local address 127.31.40.2:38411)
0928 14:59:47.110913 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:47.110923 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:47.113049 (+ 2126us) server_negotiation.cc:407] Connection header received
0928 14:59:47.113292 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:47.113301 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:47.113387 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:47.113473 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:59:47.114080 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:47.115113 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:47.117324 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:47.118083 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:47.118251 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:47.118349 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:47.118692 (+ 343us) server_negotiation.cc:299] Negotiation successful
0928 14:59:47.118837 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 14:59:47.119439 14099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40875a7e94c840fdbfa8e5dc020db5b3" candidate_uuid: "510a02d078ff4965b5dfee7317012028" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99586f213ec449ef8f1a8b1255dbb372" is_pre_election: true
I20260928 14:59:47.120080 14099 raft_consensus.cc:2454] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 510a02d078ff4965b5dfee7317012028 in term 0.
I20260928 14:59:47.120157 14231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40875a7e94c840fdbfa8e5dc020db5b3" candidate_uuid: "510a02d078ff4965b5dfee7317012028" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c34083cb0fd441ec933ad27e4602ee5e" is_pre_election: true
I20260928 14:59:47.120693 14231 raft_consensus.cc:2454] T 40875a7e94c840fdbfa8e5dc020db5b3 P c34083cb0fd441ec933ad27e4602ee5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 510a02d078ff4965b5dfee7317012028 in term 0.
I20260928 14:59:47.120774 14297 leader_election.cc:302] T 40875a7e94c840fdbfa8e5dc020db5b3 P 510a02d078ff4965b5dfee7317012028 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 510a02d078ff4965b5dfee7317012028, 99586f213ec449ef8f1a8b1255dbb372; no voters:
I20260928 14:59:47.121230 14429 raft_consensus.cc:2781] T 40875a7e94c840fdbfa8e5dc020db5b3 P 510a02d078ff4965b5dfee7317012028 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:59:47.121394 14429 raft_consensus.cc:482] T 40875a7e94c840fdbfa8e5dc020db5b3 P 510a02d078ff4965b5dfee7317012028 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:59:47.121507 14429 raft_consensus.cc:3037] T 40875a7e94c840fdbfa8e5dc020db5b3 P 510a02d078ff4965b5dfee7317012028 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:59:47.123838 14429 raft_consensus.cc:504] T 40875a7e94c840fdbfa8e5dc020db5b3 P 510a02d078ff4965b5dfee7317012028 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "510a02d078ff4965b5dfee7317012028" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 39831 } } peers { permanent_uuid: "99586f213ec449ef8f1a8b1255dbb372" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 45251 } } peers { permanent_uuid: "c34083cb0fd441ec933ad27e4602ee5e" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 38411 } }
I20260928 14:59:47.124539 14429 leader_election.cc:288] T 40875a7e94c840fdbfa8e5dc020db5b3 P 510a02d078ff4965b5dfee7317012028 [CANDIDATE]: Term 1 election: Requested vote from peers 99586f213ec449ef8f1a8b1255dbb372 (127.31.40.1:45251), c34083cb0fd441ec933ad27e4602ee5e (127.31.40.2:38411)
I20260928 14:59:47.125051 14231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40875a7e94c840fdbfa8e5dc020db5b3" candidate_uuid: "510a02d078ff4965b5dfee7317012028" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c34083cb0fd441ec933ad27e4602ee5e"
I20260928 14:59:47.125051 14099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40875a7e94c840fdbfa8e5dc020db5b3" candidate_uuid: "510a02d078ff4965b5dfee7317012028" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99586f213ec449ef8f1a8b1255dbb372"
I20260928 14:59:47.125264 14231 raft_consensus.cc:3037] T 40875a7e94c840fdbfa8e5dc020db5b3 P c34083cb0fd441ec933ad27e4602ee5e [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:59:47.125264 14099 raft_consensus.cc:3037] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:59:47.128609 14231 raft_consensus.cc:2454] T 40875a7e94c840fdbfa8e5dc020db5b3 P c34083cb0fd441ec933ad27e4602ee5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 510a02d078ff4965b5dfee7317012028 in term 1.
I20260928 14:59:47.128609 14099 raft_consensus.cc:2454] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 510a02d078ff4965b5dfee7317012028 in term 1.
I20260928 14:59:47.129037 14298 leader_election.cc:302] T 40875a7e94c840fdbfa8e5dc020db5b3 P 510a02d078ff4965b5dfee7317012028 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 510a02d078ff4965b5dfee7317012028, c34083cb0fd441ec933ad27e4602ee5e; no voters:
I20260928 14:59:47.129343 14429 raft_consensus.cc:2781] T 40875a7e94c840fdbfa8e5dc020db5b3 P 510a02d078ff4965b5dfee7317012028 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:59:47.129901 14429 raft_consensus.cc:686] T 40875a7e94c840fdbfa8e5dc020db5b3 P 510a02d078ff4965b5dfee7317012028 [term 1 LEADER]: Becoming Leader. State: Replica: 510a02d078ff4965b5dfee7317012028, State: Running, Role: LEADER
I20260928 14:59:47.130493 14429 consensus_queue.cc:237] T 40875a7e94c840fdbfa8e5dc020db5b3 P 510a02d078ff4965b5dfee7317012028 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "510a02d078ff4965b5dfee7317012028" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 39831 } } peers { permanent_uuid: "99586f213ec449ef8f1a8b1255dbb372" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 45251 } } peers { permanent_uuid: "c34083cb0fd441ec933ad27e4602ee5e" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 38411 } }
I20260928 14:59:47.135425 13959 catalog_manager.cc:5495] T 40875a7e94c840fdbfa8e5dc020db5b3 P 510a02d078ff4965b5dfee7317012028 reported cstate change: term changed from 0 to 1, leader changed from <none> to 510a02d078ff4965b5dfee7317012028 (127.31.40.3). New cstate: current_term: 1 leader_uuid: "510a02d078ff4965b5dfee7317012028" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "510a02d078ff4965b5dfee7317012028" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 39831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99586f213ec449ef8f1a8b1255dbb372" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 45251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c34083cb0fd441ec933ad27e4602ee5e" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 38411 } health_report { overall_health: UNKNOWN } } }
W20260928 14:59:47.151518 14277 tablet.cc:2369] T 40875a7e94c840fdbfa8e5dc020db5b3 P c34083cb0fd441ec933ad27e4602ee5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:59:47.154683 14408 tablet.cc:2369] T 40875a7e94c840fdbfa8e5dc020db5b3 P 510a02d078ff4965b5dfee7317012028: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:59:47.170439 14145 tablet.cc:2369] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:59:47.178454 14418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:47.172683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36992 (local address 127.31.40.3:39831)
0928 14:59:47.172962 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:47.172970 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:47.173163 (+ 193us) server_negotiation.cc:407] Connection header received
0928 14:59:47.173323 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:47.173326 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:47.173426 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:47.173559 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:59:47.173924 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:47.174624 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:47.176065 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:47.176548 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:47.176660 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:47.177151 (+ 491us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:59:47.177722 (+ 571us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:59:47.177922 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:47.178039 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 14:59:47.178117 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 14:59:47.187007 14231 raft_consensus.cc:1261] T 40875a7e94c840fdbfa8e5dc020db5b3 P c34083cb0fd441ec933ad27e4602ee5e [term 1 FOLLOWER]: Refusing update from remote peer 510a02d078ff4965b5dfee7317012028: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:59:47.187007 14099 raft_consensus.cc:1261] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372 [term 1 FOLLOWER]: Refusing update from remote peer 510a02d078ff4965b5dfee7317012028: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:59:47.188093 14433 consensus_queue.cc:1035] T 40875a7e94c840fdbfa8e5dc020db5b3 P 510a02d078ff4965b5dfee7317012028 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c34083cb0fd441ec933ad27e4602ee5e" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 38411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:59:47.188445 14429 consensus_queue.cc:1035] T 40875a7e94c840fdbfa8e5dc020db5b3 P 510a02d078ff4965b5dfee7317012028 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99586f213ec449ef8f1a8b1255dbb372" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 45251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:59:47.200682 14436 mvcc.cc:205] Tried to move back new op lower bound from 7334328677107802112 to 7334328676890390528. Current Snapshot: MvccSnapshot[applied={T|T < 7334328677107802112 or (T in {7334328677107802112})}]
I20260928 14:59:47.206511 14438 mvcc.cc:205] Tried to move back new op lower bound from 7334328677107802112 to 7334328676890390528. Current Snapshot: MvccSnapshot[applied={T|T < 7334328677107802112}]
I20260928 14:59:47.210573 13959 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:55266:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:59:47.210902 13959 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:55266:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:59:47.214709 13959 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219: Sending DeleteTablet for 3 replicas of tablet 40875a7e94c840fdbfa8e5dc020db5b3
I20260928 14:59:47.215996 14342 tablet_service.cc:1514] Processing DeleteTablet for tablet 40875a7e94c840fdbfa8e5dc020db5b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:59:47 UTC) from {username='kudu', principal='kudu/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:36988
I20260928 14:59:47.216253 14079 tablet_service.cc:1514] Processing DeleteTablet for tablet 40875a7e94c840fdbfa8e5dc020db5b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:59:47 UTC) from {username='kudu', principal='kudu/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:41244
I20260928 14:59:47.216274 14211 tablet_service.cc:1514] Processing DeleteTablet for tablet 40875a7e94c840fdbfa8e5dc020db5b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:59:47 UTC) from {username='kudu', principal='kudu/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:57648
I20260928 14:59:47.219230 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 13996
I20260928 14:59:47.226814 14445 tablet_replica.cc:330] T 40875a7e94c840fdbfa8e5dc020db5b3 P 510a02d078ff4965b5dfee7317012028: stopping tablet replica
I20260928 14:59:47.227342 14445 raft_consensus.cc:2229] T 40875a7e94c840fdbfa8e5dc020db5b3 P 510a02d078ff4965b5dfee7317012028 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:59:47.227917 14446 tablet_replica.cc:330] T 40875a7e94c840fdbfa8e5dc020db5b3 P c34083cb0fd441ec933ad27e4602ee5e: stopping tablet replica
I20260928 14:59:47.228104 14445 raft_consensus.cc:2258] T 40875a7e94c840fdbfa8e5dc020db5b3 P 510a02d078ff4965b5dfee7317012028 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:59:47.228369 14446 raft_consensus.cc:2229] T 40875a7e94c840fdbfa8e5dc020db5b3 P c34083cb0fd441ec933ad27e4602ee5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:59:47.228873 14446 raft_consensus.cc:2258] T 40875a7e94c840fdbfa8e5dc020db5b3 P c34083cb0fd441ec933ad27e4602ee5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:59:47.230578 14445 ts_tablet_manager.cc:1905] T 40875a7e94c840fdbfa8e5dc020db5b3 P 510a02d078ff4965b5dfee7317012028: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:59:47.238313 14445 ts_tablet_manager.cc:1918] T 40875a7e94c840fdbfa8e5dc020db5b3 P 510a02d078ff4965b5dfee7317012028: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:59:47.238509 14445 log.cc:1196] T 40875a7e94c840fdbfa8e5dc020db5b3 P 510a02d078ff4965b5dfee7317012028: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-2/wal/wals/40875a7e94c840fdbfa8e5dc020db5b3
I20260928 14:59:47.238701 14446 ts_tablet_manager.cc:1905] T 40875a7e94c840fdbfa8e5dc020db5b3 P c34083cb0fd441ec933ad27e4602ee5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:59:47.239197 14445 ts_tablet_manager.cc:1939] T 40875a7e94c840fdbfa8e5dc020db5b3 P 510a02d078ff4965b5dfee7317012028: Deleting consensus metadata
I20260928 14:59:47.241129 13945 catalog_manager.cc:4841] TS 510a02d078ff4965b5dfee7317012028 (127.31.40.3:39831): tablet 40875a7e94c840fdbfa8e5dc020db5b3 (table test-table [id=76201495cba041c093a151098b060349]) successfully deleted
I20260928 14:59:47.244439 14447 tablet_replica.cc:330] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372: stopping tablet replica
I20260928 14:59:47.245919 14420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:47.220983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41252 (local address 127.31.40.1:45251)
0928 14:59:47.221225 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:47.221233 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:47.221256 (+ 23us) server_negotiation.cc:407] Connection header received
0928 14:59:47.221332 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:47.221341 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:47.221441 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:47.221560 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:59:47.224349 (+ 2789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:47.229995 (+ 5646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:47.232741 (+ 2746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:47.233376 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:47.233516 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:47.237983 (+ 4467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:59:47.238006 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:59:47.238024 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:59:47.238079 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:59:47.242151 (+ 4072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:47.243124 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:47.243135 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:47.243139 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:47.243339 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:47.243781 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:47.243788 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:47.243791 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:47.244381 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:59:47.244690 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:47.245018 (+ 328us) server_negotiation.cc:299] Negotiation successful
0928 14:59:47.245265 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 14:59:47.246701 14139 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:59:47.247324 14446 ts_tablet_manager.cc:1918] T 40875a7e94c840fdbfa8e5dc020db5b3 P c34083cb0fd441ec933ad27e4602ee5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:59:47.247550 14446 log.cc:1196] T 40875a7e94c840fdbfa8e5dc020db5b3 P c34083cb0fd441ec933ad27e4602ee5e: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-1/wal/wals/40875a7e94c840fdbfa8e5dc020db5b3
I20260928 14:59:47.248256 14446 ts_tablet_manager.cc:1939] T 40875a7e94c840fdbfa8e5dc020db5b3 P c34083cb0fd441ec933ad27e4602ee5e: Deleting consensus metadata
I20260928 14:59:47.250427 13946 catalog_manager.cc:4841] TS c34083cb0fd441ec933ad27e4602ee5e (127.31.40.2:38411): tablet 40875a7e94c840fdbfa8e5dc020db5b3 (table test-table [id=76201495cba041c093a151098b060349]) successfully deleted
I20260928 14:59:47.246969 14447 raft_consensus.cc:2229] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:59:47.338984 14447 raft_consensus.cc:2258] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:59:47.339241 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 13996
I20260928 14:59:47.358809 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 14148
W20260928 14:59:47.358947 13945 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.40.1:45251 (error 108)
I20260928 14:59:47.361090 14421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:47.360494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:45251 (local address 127.0.0.1:41266)
0928 14:59:47.360779 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:47.360907 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.1:45251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
W20260928 14:59:47.361366 13945 catalog_manager.cc:4584] TS 99586f213ec449ef8f1a8b1255dbb372 (127.31.40.1:45251): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 40875a7e94c840fdbfa8e5dc020db5b3: Network error: Client connection negotiation failed: client connection to 127.31.40.1:45251: connect: Connection refused (error 111)
I20260928 14:59:47.375458 14422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:47.360003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57650 (local address 127.31.40.2:38411)
0928 14:59:47.360197 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:47.360203 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:47.360224 (+ 21us) server_negotiation.cc:407] Connection header received
0928 14:59:47.360344 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:47.360350 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:47.360436 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:47.360558 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:59:47.362223 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:47.363350 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:47.364901 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:47.365447 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:47.365578 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:47.368207 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:59:47.368226 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:59:47.368249 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:59:47.368293 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:59:47.372007 (+ 3714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:47.372928 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:47.372939 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:47.372943 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:47.373090 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:47.373519 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:47.373524 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:47.373528 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:47.374068 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:59:47.374359 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:47.374690 (+ 331us) server_negotiation.cc:299] Negotiation successful
0928 14:59:47.374924 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 14:59:47.376521 14271 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:59:47.415246 14421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:47.414719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:45251 (local address 127.0.0.1:41272)
0928 14:59:47.414935 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:47.415040 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.1:45251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260928 14:59:47.416502 14421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:47.416084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:45251 (local address 127.0.0.1:41282)
0928 14:59:47.416307 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:47.416384 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.1:45251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260928 14:59:47.477187 14421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:47.476601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:45251 (local address 127.0.0.1:41298)
0928 14:59:47.476847 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:47.476973 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.1:45251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260928 14:59:47.478447 14421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:47.478027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:45251 (local address 127.0.0.1:41306)
0928 14:59:47.478207 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:47.478302 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.1:45251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 14:59:47.504634 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 14148
I20260928 14:59:47.528532 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 14279
I20260928 14:59:47.546265 14418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:47.529420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37000 (local address 127.31.40.3:39831)
0928 14:59:47.529602 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:47.529607 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:47.529634 (+ 27us) server_negotiation.cc:407] Connection header received
0928 14:59:47.529860 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:47.529864 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:47.529998 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:47.530132 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:59:47.531960 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:47.532724 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:47.534645 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:47.535127 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:47.535232 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:47.539011 (+ 3779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:59:47.539023 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:59:47.539039 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:59:47.539084 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:59:47.542516 (+ 3432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:47.543539 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:47.543547 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:47.543549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:47.543726 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:47.544445 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:47.544449 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:47.544451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:47.544847 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:59:47.545144 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:47.545554 (+ 410us) server_negotiation.cc:299] Negotiation successful
0928 14:59:47.545724 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 14:59:47.547130 14402 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:59:47.559170 14421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:47.558695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:45251 (local address 127.0.0.1:41318)
0928 14:59:47.558913 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:47.559016 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.1:45251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260928 14:59:47.560271 14421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:47.559806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:45251 (local address 127.0.0.1:41330)
0928 14:59:47.560079 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:47.560164 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.1:45251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168}
I20260928 14:59:47.672127 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 14279
I20260928 14:59:47.694909 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 13926
I20260928 14:59:47.696168 13989 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 14:59:47.812726 14421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:47.812005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:45251 (local address 127.0.0.1:41338)
0928 14:59:47.812232 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:47.812394 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.1:45251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260928 14:59:47.812886 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 13926
I20260928 14:59:47.836122 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:43875
--webserver_interface=127.31.40.62
--webserver_port=36845
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:39673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.40.62:43875
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:59:48.173753 14453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:59:48.174109 14453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:59:48.174261 14453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:59:48.181741 14453 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:59:48.181921 14453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:59:48.182029 14453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:59:48.182137 14453 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:59:48.182214 14453 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:59:48.192711 14453 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:39673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.40.62:43875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:43875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.40.62
--webserver_port=36845
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:59:48.194172 14453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:59:48.195531 14453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:59:48.208135 14458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:59:48.208105 14459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:59:48.208191 14461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:59:48.209277 14453 server_base.cc:1034] running on GCE node
I20260928 14:59:48.210075 14453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:59:48.211565 14453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:59:48.212805 14453 hybrid_clock.cc:648] HybridClock initialized: now 1790607588212755 us; error 34 us; skew 500 ppm
I20260928 14:59:48.213269 14453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:59:48 dist-test-slave-j968 krb5kdc[13374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607588, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:59:48.216711 14453 init.cc:377] Logged in from keytab as kudu/127.31.40.62@KRBTEST.COM (short username kudu)
I20260928 14:59:48.218417 14453 webserver.cc:460] Webserver started at http://127.31.40.62:36845/ using document root <none> and password file <none>
I20260928 14:59:48.219151 14453 fs_manager.cc:359] Metadata directory not provided
I20260928 14:59:48.219291 14453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:59:48.225247 14453 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 14:59:48.228513 14468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:59:48.230079 14453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:59:48.230269 14453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/master-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/master-0/wal
uuid: "40864fbc798f4c4b8033a199f800a219"
format_stamp: "Formatted at 2026-09-28 14:59:42 on dist-test-slave-j968"
server_key: "463f55961d213e59e4bd0704c14b73f7"
server_key_iv: "527ed252030efd282a04fd5f1b04591d"
server_key_version: "encryptionkey@0"
I20260928 14:59:48.231709 14453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:59:48.245847 14453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:59:48.250389 14453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:59:48.250624 14453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:59:48.269622 14453 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.62:43875
I20260928 14:59:48.269642 14519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.62:43875 every 8 connection(s)
I20260928 14:59:48.271150 14453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
I20260928 14:59:48.278267 14520 sys_catalog.cc:263] Verifying existing consensus state
I20260928 14:59:48.279572 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 14453
I20260928 14:59:48.280592 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.1:45251
--local_ip_for_outbound_sockets=127.31.40.1
--tserver_master_addrs=127.31.40.62:43875
--webserver_port=36631
--webserver_interface=127.31.40.1
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab
--principal=kudu/127.31.40.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:39673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:59:48.283105 14520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219: Bootstrap starting.
I20260928 14:59:48.305557 14520 log.cc:824] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219: Log is configured to *not* fsync() on all Append() calls
I20260928 14:59:48.322643 14520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:59:48.323231 14520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219: Bootstrap complete.
I20260928 14:59:48.332811 14520 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40864fbc798f4c4b8033a199f800a219" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 43875 } }
I20260928 14:59:48.333722 14520 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40864fbc798f4c4b8033a199f800a219, State: Initialized, Role: FOLLOWER
I20260928 14:59:48.334388 14520 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40864fbc798f4c4b8033a199f800a219" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 43875 } }
I20260928 14:59:48.334636 14520 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:59:48.334751 14520 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:59:48.334947 14520 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219 [term 2 FOLLOWER]: Advancing to term 3
I20260928 14:59:48.339679 14520 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40864fbc798f4c4b8033a199f800a219" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 43875 } }
I20260928 14:59:48.340236 14520 leader_election.cc:302] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40864fbc798f4c4b8033a199f800a219; no voters:
I20260928 14:59:48.340930 14520 leader_election.cc:288] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 14:59:48.341080 14524 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 14:59:48.343346 14520 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:59:48.344165 14524 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219 [term 3 LEADER]: Becoming Leader. State: Replica: 40864fbc798f4c4b8033a199f800a219, State: Running, Role: LEADER
I20260928 14:59:48.344734 14524 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40864fbc798f4c4b8033a199f800a219" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 43875 } }
I20260928 14:59:48.347625 14526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40864fbc798f4c4b8033a199f800a219. Latest consensus state: current_term: 3 leader_uuid: "40864fbc798f4c4b8033a199f800a219" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40864fbc798f4c4b8033a199f800a219" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 43875 } } }
I20260928 14:59:48.347652 14525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "40864fbc798f4c4b8033a199f800a219" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40864fbc798f4c4b8033a199f800a219" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 43875 } } }
I20260928 14:59:48.348097 14526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219 [sys.catalog]: This master's current role is: LEADER
I20260928 14:59:48.348130 14525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40864fbc798f4c4b8033a199f800a219 [sys.catalog]: This master's current role is: LEADER
I20260928 14:59:48.359210 14534 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:59:48.367736 14534 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:59:48.369266 14534 catalog_manager.cc:1259] Loaded cluster ID: dd0d8e389e114f4fbaf6eb4039ae60a7
I20260928 14:59:48.369376 14534 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 14:59:48.371606 14534 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f81d2c8e9ad google::LogMessage::Fail() at ??:0
@ 0x7f81d2c9188a google::LogMessage::SendToLog() at ??:0
@ 0x7f81d2c8e4de google::LogMessage::Flush() at ??:0
@ 0x7f81d2c8fa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f81e86c1512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f81e86be976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f81e8746610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f81e8746439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f81d58da345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f81d58dd91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f81d58dd799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f81d58b8eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f81d8b926db start_thread at ??:0
@ 0x7f81d1b2861f clone at ??:0
W20260928 14:59:48.660715 14522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:59:48.661084 14522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:59:48.661254 14522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:59:48.669039 14522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:59:48.669191 14522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:59:48.669369 14522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.1
I20260928 14:59:48.679643 14522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:39673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab
--principal=kudu/127.31.40.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.1:45251
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.40.1
--webserver_port=36631
--tserver_master_addrs=127.31.40.62:43875
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.1
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:59:48.681113 14522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:59:48.682499 14522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:59:48.697043 14562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:59:48.697057 14561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:59:48.697232 14564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:59:48.697751 14522 server_base.cc:1034] running on GCE node
I20260928 14:59:48.698426 14522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:59:48.699432 14522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:59:48.700685 14522 hybrid_clock.cc:648] HybridClock initialized: now 1790607588700606 us; error 65 us; skew 500 ppm
I20260928 14:59:48.701061 14522 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:59:48 dist-test-slave-j968 krb5kdc[13374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607588, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:59:48.704322 14522 init.cc:377] Logged in from keytab as kudu/127.31.40.1@KRBTEST.COM (short username kudu)
I20260928 14:59:48.705931 14522 webserver.cc:460] Webserver started at http://127.31.40.1:36631/ using document root <none> and password file <none>
I20260928 14:59:48.706732 14522 fs_manager.cc:359] Metadata directory not provided
I20260928 14:59:48.706883 14522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:59:48.712455 14522 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 14:59:48.715390 14571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:59:48.717000 14522 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:59:48.717180 14522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-0/wal
uuid: "99586f213ec449ef8f1a8b1255dbb372"
format_stamp: "Formatted at 2026-09-28 14:59:42 on dist-test-slave-j968"
server_key: "b0b4b056fd8add989e3eb7e289374c91"
server_key_iv: "a9788ed67f36beaa5d647573f7b9802c"
server_key_version: "encryptionkey@0"
I20260928 14:59:48.718178 14522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:59:48.737021 14522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:59:48.741482 14522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:59:48.741791 14522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:59:48.742846 14522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:59:48.744762 14578 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260928 14:59:48.750638 14522 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 14:59:48.750792 14522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.001s
I20260928 14:59:48.750948 14522 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 14:59:48.754606 14522 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 14:59:48.754736 14522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.000s sys 0.001s
I20260928 14:59:48.755214 14578 tablet_bootstrap.cc:492] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372: Bootstrap starting.
I20260928 14:59:48.784379 14578 log.cc:824] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372: Log is configured to *not* fsync() on all Append() calls
I20260928 14:59:48.799978 14522 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.1:45251
I20260928 14:59:48.800031 14685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.1:45251 every 8 connection(s)
I20260928 14:59:48.801278 14578 tablet_bootstrap.cc:492] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:59:48.801328 14522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
I20260928 14:59:48.801864 14578 tablet_bootstrap.cc:492] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372: Bootstrap complete.
I20260928 14:59:48.803608 14578 ts_tablet_manager.cc:1397] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372: Time spent bootstrapping tablet: real 0.049s user 0.026s sys 0.020s
I20260928 14:59:48.807387 14688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:48.804815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.1:43961)
0928 14:59:48.805751 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:48.806078 (+ 327us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:43875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":384,"threads_started":1}
W20260928 14:59:48.808292 14686 heartbeater.cc:644] Failed to heartbeat to 127.31.40.62:43875 (0 consecutive failures): Network error: Failed to ping master at 127.31.40.62:43875: Client connection negotiation failed: client connection to 127.31.40.62:43875: connect: Connection refused (error 111)
I20260928 14:59:48.809214 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 14522
I20260928 14:59:48.809271 14688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:48.808860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.1:39625)
0928 14:59:48.809081 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:48.809158 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:43875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260928 14:59:48.810025 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.2:38411
--local_ip_for_outbound_sockets=127.31.40.2
--tserver_master_addrs=127.31.40.62:43875
--webserver_port=39369
--webserver_interface=127.31.40.2
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab
--principal=kudu/127.31.40.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:39673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:59:48.810364 14688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:48.809936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.1:37149)
0928 14:59:48.810087 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:48.810198 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:43875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 14:59:48.810681 14686 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 14:59:48.814028 14578 raft_consensus.cc:348] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "510a02d078ff4965b5dfee7317012028" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 39831 } } peers { permanent_uuid: "99586f213ec449ef8f1a8b1255dbb372" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 45251 } } peers { permanent_uuid: "c34083cb0fd441ec933ad27e4602ee5e" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 38411 } }
I20260928 14:59:48.815562 14578 raft_consensus.cc:729] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99586f213ec449ef8f1a8b1255dbb372, State: Initialized, Role: FOLLOWER
I20260928 14:59:48.816378 14578 consensus_queue.cc:260] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "510a02d078ff4965b5dfee7317012028" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 39831 } } peers { permanent_uuid: "99586f213ec449ef8f1a8b1255dbb372" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 45251 } } peers { permanent_uuid: "c34083cb0fd441ec933ad27e4602ee5e" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 38411 } }
I20260928 14:59:48.818163 14688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:48.817782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.1:43701)
0928 14:59:48.817947 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:48.818037 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:43875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 14:59:48.818338 14578 ts_tablet_manager.cc:1428] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260928 14:59:48.819703 14688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:48.819241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.1:41991)
0928 14:59:48.819430 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:48.819517 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:43875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260928 14:59:49.154855 14690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:59:49.155252 14690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:59:49.155441 14690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:59:49.163074 14690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:59:49.163267 14690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:59:49.163462 14690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.2
I20260928 14:59:49.173775 14690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:39673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab
--principal=kudu/127.31.40.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.2:38411
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.40.2
--webserver_port=39369
--tserver_master_addrs=127.31.40.62:43875
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.2
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:59:49.175174 14690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:59:49.176652 14690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:59:49.190881 14696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:59:49.190980 14695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:59:49.190904 14698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:59:49.192142 14690 server_base.cc:1034] running on GCE node
I20260928 14:59:49.192781 14690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:59:49.193935 14690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:59:49.195207 14690 hybrid_clock.cc:648] HybridClock initialized: now 1790607589195146 us; error 40 us; skew 500 ppm
I20260928 14:59:49.195534 14690 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:59:49 dist-test-slave-j968 krb5kdc[13374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607589, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:59:49.199203 14690 init.cc:377] Logged in from keytab as kudu/127.31.40.2@KRBTEST.COM (short username kudu)
I20260928 14:59:49.200799 14690 webserver.cc:460] Webserver started at http://127.31.40.2:39369/ using document root <none> and password file <none>
I20260928 14:59:49.201548 14690 fs_manager.cc:359] Metadata directory not provided
I20260928 14:59:49.201685 14690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:59:49.207223 14690 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 14:59:49.210194 14705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:59:49.211588 14690 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:59:49.211771 14690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-1/wal
uuid: "c34083cb0fd441ec933ad27e4602ee5e"
format_stamp: "Formatted at 2026-09-28 14:59:43 on dist-test-slave-j968"
server_key: "21ed01889dc007b0666c606660d5b18e"
server_key_iv: "7d60ad8a36b4fedaac4945143323055b"
server_key_version: "encryptionkey@0"
I20260928 14:59:49.212838 14690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:59:49.237717 14690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:59:49.242197 14690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:59:49.242519 14690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:59:49.243549 14690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:59:49.245014 14690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:59:49.245143 14690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:59:49.245273 14690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:59:49.245359 14690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:59:49.278712 14690 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.2:38411
I20260928 14:59:49.278760 14817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.2:38411 every 8 connection(s)
I20260928 14:59:49.280258 14690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
I20260928 14:59:49.285149 14820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:49.283258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.2:47243)
0928 14:59:49.284057 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:49.284271 (+ 214us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:43875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":267,"threads_started":1}
W20260928 14:59:49.285825 14818 heartbeater.cc:644] Failed to heartbeat to 127.31.40.62:43875 (0 consecutive failures): Network error: Failed to ping master at 127.31.40.62:43875: Client connection negotiation failed: client connection to 127.31.40.62:43875: connect: Connection refused (error 111)
I20260928 14:59:49.286626 14820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:49.286307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.2:55373)
0928 14:59:49.286460 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:49.286526 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:43875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 14:59:49.287542 14820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:49.287237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.2:47149)
0928 14:59:49.287355 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:49.287427 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:43875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 14:59:49.287906 14818 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 14:59:49.288573 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 14690
I20260928 14:59:49.289173 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.3:39831
--local_ip_for_outbound_sockets=127.31.40.3
--tserver_master_addrs=127.31.40.62:43875
--webserver_port=43059
--webserver_interface=127.31.40.3
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab
--principal=kudu/127.31.40.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:39673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:59:49.627646 14821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:59:49.628022 14821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:59:49.628125 14821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:59:49.635382 14821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:59:49.635504 14821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:59:49.635619 14821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.3
I20260928 14:59:49.645773 14821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:39673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab
--principal=kudu/127.31.40.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.3:39831
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.40.3
--webserver_port=43059
--tserver_master_addrs=127.31.40.62:43875
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.3
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 14:59:49.647066 14821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:59:49.648468 14821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:59:49.662398 14826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:59:49.662369 14829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:59:49.662465 14827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:59:49.663038 14821 server_base.cc:1034] running on GCE node
I20260928 14:59:49.663611 14821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:59:49.664577 14821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:59:49.665776 14821 hybrid_clock.cc:648] HybridClock initialized: now 1790607589665705 us; error 60 us; skew 500 ppm
I20260928 14:59:49.666062 14821 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 14:59:49 dist-test-slave-j968 krb5kdc[13374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607589, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:59:49.669384 14821 init.cc:377] Logged in from keytab as kudu/127.31.40.3@KRBTEST.COM (short username kudu)
I20260928 14:59:49.670935 14821 webserver.cc:460] Webserver started at http://127.31.40.3:43059/ using document root <none> and password file <none>
I20260928 14:59:49.671643 14821 fs_manager.cc:359] Metadata directory not provided
I20260928 14:59:49.671814 14821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:59:49.677201 14821 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:59:49.680213 14836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:59:49.681543 14821 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:59:49.681725 14821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-2/wal
uuid: "510a02d078ff4965b5dfee7317012028"
format_stamp: "Formatted at 2026-09-28 14:59:43 on dist-test-slave-j968"
server_key: "67992f9ae5900758844c16c37b5a00d6"
server_key_iv: "194cf94a9a7262c84a9b9ae52218bd35"
server_key_version: "encryptionkey@0"
I20260928 14:59:49.682740 14821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:59:49.699612 14821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:59:49.704219 14821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:59:49.704509 14821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:59:49.705492 14821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:59:49.706807 14821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:59:49.706966 14821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:59:49.707095 14821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:59:49.707192 14821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:59:49.742252 14821 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.3:39831
I20260928 14:59:49.742297 14948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.3:39831 every 8 connection(s)
I20260928 14:59:49.743817 14821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
I20260928 14:59:49.744225 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 14821
I20260928 14:59:49.749028 14952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:49.747169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.3:38783)
0928 14:59:49.747936 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:49.748138 (+ 202us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:43875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":278,"threads_started":1}
W20260928 14:59:49.749819 14949 heartbeater.cc:644] Failed to heartbeat to 127.31.40.62:43875 (0 consecutive failures): Network error: Failed to ping master at 127.31.40.62:43875: Client connection negotiation failed: client connection to 127.31.40.62:43875: connect: Connection refused (error 111)
I20260928 14:59:49.750623 14952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:49.750312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.3:55165)
0928 14:59:49.750454 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:49.750519 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:43875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 14:59:49.751433 14952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:49.751141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.3:52507)
0928 14:59:49.751261 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:49.751327 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:43875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 14:59:49.751708 14949 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 14:59:49.821389 14953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:49.820564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.1:53179)
0928 14:59:49.821062 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:49.821175 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:43875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":242,"threads_started":1}
I20260928 14:59:50.181344 14954 raft_consensus.cc:482] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:59:50.181624 14954 raft_consensus.cc:504] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "510a02d078ff4965b5dfee7317012028" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 39831 } } peers { permanent_uuid: "99586f213ec449ef8f1a8b1255dbb372" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 45251 } } peers { permanent_uuid: "c34083cb0fd441ec933ad27e4602ee5e" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 38411 } }
I20260928 14:59:50.183324 14954 leader_election.cc:288] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 510a02d078ff4965b5dfee7317012028 (127.31.40.3:39831), c34083cb0fd441ec933ad27e4602ee5e (127.31.40.2:38411)
Sep 28 14:59:50 dist-test-slave-j968 krb5kdc[13374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607588, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.1@KRBTEST.COM for kudu/127.31.40.2@KRBTEST.COM
Sep 28 14:59:50 dist-test-slave-j968 krb5kdc[13374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607588, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.1@KRBTEST.COM for kudu/127.31.40.3@KRBTEST.COM
I20260928 14:59:50.205791 14953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:50.183702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.3:39831 (local address 127.31.40.1:50809)
0928 14:59:50.184247 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:50.184311 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 14:59:50.184885 (+ 574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:59:50.188330 (+ 3445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:59:50.188341 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:59:50.189111 (+ 770us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:59:50.189826 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:50.189838 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:50.191451 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:50.191454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:50.192958 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:50.192970 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:50.194024 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:50.194030 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:50.194356 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:50.195195 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:59:50.195219 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:59:50.198363 (+ 3144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:59:50.202058 (+ 3695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:59:50.202067 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:59:50.202071 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:59:50.202570 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:59:50.203057 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:59:50.203064 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:59:50.203067 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:59:50.203213 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:59:50.204324 (+ 1111us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:59:50.204335 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:59:50.204778 (+ 443us) client_negotiation.cc:769] Sending connection context
0928 14:59:50.205011 (+ 233us) client_negotiation.cc:240] Negotiation successful
0928 14:59:50.205447 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":219,"spinlock_wait_cycles":35456}
I20260928 14:59:50.206180 14955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:50.183935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.2:38411 (local address 127.31.40.1:42349)
0928 14:59:50.184971 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:50.184993 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 14:59:50.185126 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:59:50.188330 (+ 3204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:59:50.188341 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:59:50.189103 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:59:50.189771 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:50.189795 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:50.191416 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:50.191421 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:50.192953 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:50.192966 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:50.193985 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:50.193992 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:50.194351 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:50.195184 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:59:50.195215 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:59:50.198204 (+ 2989us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:59:50.201981 (+ 3777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:59:50.201992 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:59:50.202010 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:59:50.202556 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:59:50.203009 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:59:50.203015 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:59:50.203018 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:59:50.203206 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:59:50.204337 (+ 1131us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:59:50.204343 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:59:50.204759 (+ 416us) client_negotiation.cc:769] Sending connection context
0928 14:59:50.205045 (+ 286us) client_negotiation.cc:240] Negotiation successful
0928 14:59:50.205639 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":884,"mutex_wait_us":340,"spinlock_wait_cycles":10624,"thread_start_us":833,"threads_started":1}
I20260928 14:59:50.206179 14956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:50.184149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:42349 (local address 127.31.40.2:38411)
0928 14:59:50.185607 (+ 1458us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:50.185618 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:50.185646 (+ 28us) server_negotiation.cc:407] Connection header received
0928 14:59:50.187332 (+ 1686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:50.187341 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:50.187767 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:50.188155 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:59:50.189989 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:50.191208 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:50.193114 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:50.193768 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:50.193964 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:50.198561 (+ 4597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:59:50.198570 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:59:50.198586 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:59:50.198625 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:59:50.201789 (+ 3164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:50.202708 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:50.202721 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:50.202726 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:50.202880 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:50.203329 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:50.203334 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:50.203337 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:50.204063 (+ 726us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:59:50.204263 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:50.205287 (+ 1024us) server_negotiation.cc:299] Negotiation successful
0928 14:59:50.205712 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1306,"thread_start_us":1187,"threads_started":1}
I20260928 14:59:50.208374 14957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:50.184397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:50809 (local address 127.31.40.3:39831)
0928 14:59:50.185370 (+ 973us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:50.185383 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:50.185416 (+ 33us) server_negotiation.cc:407] Connection header received
0928 14:59:50.187341 (+ 1925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:50.187350 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:50.187745 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:50.188173 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:59:50.190064 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:50.191237 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:50.193109 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:50.193776 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:50.193999 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:50.198626 (+ 4627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:59:50.198634 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:59:50.198646 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:59:50.198687 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:59:50.201912 (+ 3225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:50.202744 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:50.202750 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:50.202756 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:50.202908 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:50.203355 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:50.203359 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:50.203361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:50.204067 (+ 706us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:59:50.204252 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:50.207661 (+ 3409us) server_negotiation.cc:299] Negotiation successful
0928 14:59:50.208091 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":805,"thread_start_us":671,"threads_started":1}
W20260928 14:59:50.209501 14575 leader_election.cc:341] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c34083cb0fd441ec933ad27e4602ee5e (127.31.40.2:38411): Not found: Tablet not found: 40875a7e94c840fdbfa8e5dc020db5b3
W20260928 14:59:50.210981 14574 leader_election.cc:341] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 510a02d078ff4965b5dfee7317012028 (127.31.40.3:39831): Not found: Tablet not found: 40875a7e94c840fdbfa8e5dc020db5b3
I20260928 14:59:50.211102 14574 leader_election.cc:302] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99586f213ec449ef8f1a8b1255dbb372; no voters: 510a02d078ff4965b5dfee7317012028, c34083cb0fd441ec933ad27e4602ee5e
I20260928 14:59:50.211467 14954 raft_consensus.cc:2726] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 14:59:50.289319 14958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:50.288435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.2:40323)
0928 14:59:50.288929 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:50.289075 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:43875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":245,"threads_started":1}
I20260928 14:59:50.753077 14959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:50.752325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.3:60361)
0928 14:59:50.752763 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:50.752868 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:43875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":219,"threads_started":1}
I20260928 14:59:50.823302 14960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:50.822437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.1:47817)
0928 14:59:50.823034 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:50.823123 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:43875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":338,"threads_started":1}
I20260928 14:59:51.291152 14961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:51.290340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.2:56993)
0928 14:59:51.290882 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:51.290995 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:43875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":302,"threads_started":1}
I20260928 14:59:51.754736 14962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:51.754045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.3:58767)
0928 14:59:51.754465 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:51.754555 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:43875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":226,"threads_started":1}
I20260928 14:59:51.825270 14963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:51.824379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.1:56551)
0928 14:59:51.824959 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:51.825054 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:43875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":241,"threads_started":1}
I20260928 14:59:52.160275 14964 raft_consensus.cc:482] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:59:52.160468 14964 raft_consensus.cc:504] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "510a02d078ff4965b5dfee7317012028" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 39831 } } peers { permanent_uuid: "99586f213ec449ef8f1a8b1255dbb372" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 45251 } } peers { permanent_uuid: "c34083cb0fd441ec933ad27e4602ee5e" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 38411 } }
I20260928 14:59:52.161114 14964 leader_election.cc:288] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 510a02d078ff4965b5dfee7317012028 (127.31.40.3:39831), c34083cb0fd441ec933ad27e4602ee5e (127.31.40.2:38411)
W20260928 14:59:52.162237 14575 leader_election.cc:341] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c34083cb0fd441ec933ad27e4602ee5e (127.31.40.2:38411): Not found: Tablet not found: 40875a7e94c840fdbfa8e5dc020db5b3
W20260928 14:59:52.162424 14574 leader_election.cc:341] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 510a02d078ff4965b5dfee7317012028 (127.31.40.3:39831): Not found: Tablet not found: 40875a7e94c840fdbfa8e5dc020db5b3
I20260928 14:59:52.162572 14574 leader_election.cc:302] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99586f213ec449ef8f1a8b1255dbb372; no voters: 510a02d078ff4965b5dfee7317012028, c34083cb0fd441ec933ad27e4602ee5e
I20260928 14:59:52.162873 14964 raft_consensus.cc:2726] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 14:59:52.292892 14965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:52.292060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.2:48307)
0928 14:59:52.292540 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:52.292675 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:43875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":249,"threads_started":1}
I20260928 14:59:52.756500 14966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:52.755664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.3:49301)
0928 14:59:52.756212 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:52.756317 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:43875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":291,"threads_started":1}
I20260928 14:59:52.827112 14967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:52.826376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.1:51537)
0928 14:59:52.826836 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:52.826932 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:43875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":231,"threads_started":1}
I20260928 14:59:53.294524 14968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:53.293811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.2:42339)
0928 14:59:53.294259 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:53.294341 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:43875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":228,"threads_started":1}
I20260928 14:59:53.758543 14969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:53.757470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.3:56423)
0928 14:59:53.758148 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:53.758258 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:43875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":252,"threads_started":1}
I20260928 14:59:53.829190 14970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:53.828409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.1:60721)
0928 14:59:53.828903 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:53.828991 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:43875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":220,"threads_started":1}
I20260928 14:59:53.986701 14971 raft_consensus.cc:482] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:59:53.987023 14971 raft_consensus.cc:504] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "510a02d078ff4965b5dfee7317012028" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 39831 } } peers { permanent_uuid: "99586f213ec449ef8f1a8b1255dbb372" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 45251 } } peers { permanent_uuid: "c34083cb0fd441ec933ad27e4602ee5e" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 38411 } }
I20260928 14:59:53.987739 14971 leader_election.cc:288] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 510a02d078ff4965b5dfee7317012028 (127.31.40.3:39831), c34083cb0fd441ec933ad27e4602ee5e (127.31.40.2:38411)
W20260928 14:59:53.988658 14574 leader_election.cc:341] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 510a02d078ff4965b5dfee7317012028 (127.31.40.3:39831): Not found: Tablet not found: 40875a7e94c840fdbfa8e5dc020db5b3
W20260928 14:59:53.988816 14575 leader_election.cc:341] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c34083cb0fd441ec933ad27e4602ee5e (127.31.40.2:38411): Not found: Tablet not found: 40875a7e94c840fdbfa8e5dc020db5b3
I20260928 14:59:53.988915 14575 leader_election.cc:302] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99586f213ec449ef8f1a8b1255dbb372; no voters: 510a02d078ff4965b5dfee7317012028, c34083cb0fd441ec933ad27e4602ee5e
I20260928 14:59:53.989176 14971 raft_consensus.cc:2726] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 14:59:54.296166 14972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:54.295454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.2:38435)
0928 14:59:54.295924 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:54.296007 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:43875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":219,"threads_started":1}
I20260928 14:59:54.760452 14973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:54.759680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.3:43103)
0928 14:59:54.760173 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:54.760265 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:43875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":295,"threads_started":1}
I20260928 14:59:54.831130 14974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:54.830252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.1:43527)
0928 14:59:54.830763 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:54.830941 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:43875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":297,"threads_started":1}
I20260928 14:59:55.297916 14975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:55.297103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.2:54431)
0928 14:59:55.297644 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:55.297722 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:43875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":306,"threads_started":1}
I20260928 14:59:55.762126 14976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:55.761380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.3:46369)
0928 14:59:55.761855 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:55.761945 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:43875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":238,"threads_started":1}
I20260928 14:59:55.832871 14977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:55.832118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.1:55741)
0928 14:59:55.832579 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:55.832667 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:43875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":249,"threads_started":1}
I20260928 14:59:56.299786 14978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:56.298922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.2:53985)
0928 14:59:56.299498 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:56.299591 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:43875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":250,"threads_started":1}
I20260928 14:59:56.764175 14979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:56.763245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.3:50777)
0928 14:59:56.763766 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:56.763890 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:43875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":239,"threads_started":1}
I20260928 14:59:56.835018 14980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:56.834151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.1:44239)
0928 14:59:56.834719 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:56.834815 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:43875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":314,"threads_started":1}
I20260928 14:59:57.301764 14981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:57.300945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.2:41211)
0928 14:59:57.301468 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:57.301558 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:43875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":298,"threads_started":1}
I20260928 14:59:57.765970 14982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:57.765186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.3:48971)
0928 14:59:57.765628 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:57.765719 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:43875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":231,"threads_started":1}
I20260928 14:59:57.836795 14983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:57.836039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.1:55053)
0928 14:59:57.836512 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:57.836607 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:43875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":231,"threads_started":1}
I20260928 14:59:58.303946 14984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:58.302960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.2:47359)
0928 14:59:58.303568 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:58.303686 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:43875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":284,"threads_started":1}
I20260928 14:59:58.767795 14985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:58.767014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.3:34353)
0928 14:59:58.767540 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:58.767632 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:43875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":251,"threads_started":1}
I20260928 14:59:58.775223 14986 raft_consensus.cc:482] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:59:58.775444 14986 raft_consensus.cc:504] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "510a02d078ff4965b5dfee7317012028" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 39831 } } peers { permanent_uuid: "99586f213ec449ef8f1a8b1255dbb372" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 45251 } } peers { permanent_uuid: "c34083cb0fd441ec933ad27e4602ee5e" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 38411 } }
I20260928 14:59:58.776278 14986 leader_election.cc:288] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 510a02d078ff4965b5dfee7317012028 (127.31.40.3:39831), c34083cb0fd441ec933ad27e4602ee5e (127.31.40.2:38411)
W20260928 14:59:58.777408 14574 leader_election.cc:341] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 510a02d078ff4965b5dfee7317012028 (127.31.40.3:39831): Not found: Tablet not found: 40875a7e94c840fdbfa8e5dc020db5b3
W20260928 14:59:58.777644 14575 leader_election.cc:341] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c34083cb0fd441ec933ad27e4602ee5e (127.31.40.2:38411): Not found: Tablet not found: 40875a7e94c840fdbfa8e5dc020db5b3
I20260928 14:59:58.777957 14575 leader_election.cc:302] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99586f213ec449ef8f1a8b1255dbb372; no voters: 510a02d078ff4965b5dfee7317012028, c34083cb0fd441ec933ad27e4602ee5e
I20260928 14:59:58.778384 14986 raft_consensus.cc:2726] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 14:59:58.839008 14987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:58.838105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.1:37661)
0928 14:59:58.838651 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:58.838771 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:43875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":292,"threads_started":1}
I20260928 14:59:59.305958 14988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:59.305083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.2:46331)
0928 14:59:59.305683 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:59.305766 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:43875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":385,"threads_started":1}
I20260928 14:59:59.769819 14989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:59.768958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.3:41311)
0928 14:59:59.769464 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:59.769561 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:43875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":277,"threads_started":1}
I20260928 14:59:59.840821 14990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:59.840106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.1:44615)
0928 14:59:59.840547 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:59.840625 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:43875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":231,"threads_started":1}
I20260928 15:00:00.307824 14991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:00.306987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.2:33223)
0928 15:00:00.307569 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:00.307661 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:43875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":318,"threads_started":1}
I20260928 15:00:00.771585 14992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:00.770835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.3:54423)
0928 15:00:00.771315 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:00.771420 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:43875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":243,"threads_started":1}
I20260928 15:00:00.842577 14993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:00.841830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.1:53607)
0928 15:00:00.842299 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:00.842383 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:43875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":244,"threads_started":1}
I20260928 15:00:01.309834 14994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:01.308966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.2:51375)
0928 15:00:01.309529 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:01.309629 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:43875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":326,"threads_started":1}
I20260928 15:00:01.773522 14995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:01.772679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.3:60979)
0928 15:00:01.773235 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:01.773314 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:43875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":283,"threads_started":1}
I20260928 15:00:01.844672 14996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:01.843585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.1:60371)
0928 15:00:01.844355 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:01.844470 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:43875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":484,"threads_started":1}
I20260928 15:00:02.311749 14997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:02.310949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.2:44307)
0928 15:00:02.311469 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:02.311573 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:43875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":252,"threads_started":1}
I20260928 15:00:02.775280 14998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:02.774550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.3:54773)
0928 15:00:02.775029 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:02.775116 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:43875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":224,"threads_started":1}
I20260928 15:00:02.846417 14999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:02.845644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.1:42377)
0928 15:00:02.846138 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:02.846224 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:43875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":261,"threads_started":1}
I20260928 15:00:03.314016 15000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:03.313016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.2:60803)
0928 15:00:03.313703 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:03.313794 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:43875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":351,"threads_started":1}
I20260928 15:00:03.777014 15001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:03.776230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.3:33169)
0928 15:00:03.776720 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:03.776813 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:43875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":256,"threads_started":1}
I20260928 15:00:03.848626 15002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:03.847725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.1:40419)
0928 15:00:03.848302 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:03.848435 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:43875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":316,"threads_started":1}
I20260928 15:00:04.238966 15003 raft_consensus.cc:482] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:00:04.239166 15003 raft_consensus.cc:504] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "510a02d078ff4965b5dfee7317012028" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 39831 } } peers { permanent_uuid: "99586f213ec449ef8f1a8b1255dbb372" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 45251 } } peers { permanent_uuid: "c34083cb0fd441ec933ad27e4602ee5e" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 38411 } }
I20260928 15:00:04.240003 15003 leader_election.cc:288] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 510a02d078ff4965b5dfee7317012028 (127.31.40.3:39831), c34083cb0fd441ec933ad27e4602ee5e (127.31.40.2:38411)
W20260928 15:00:04.241035 14574 leader_election.cc:341] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 510a02d078ff4965b5dfee7317012028 (127.31.40.3:39831): Not found: Tablet not found: 40875a7e94c840fdbfa8e5dc020db5b3
W20260928 15:00:04.241272 14575 leader_election.cc:341] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c34083cb0fd441ec933ad27e4602ee5e (127.31.40.2:38411): Not found: Tablet not found: 40875a7e94c840fdbfa8e5dc020db5b3
I20260928 15:00:04.241420 14575 leader_election.cc:302] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99586f213ec449ef8f1a8b1255dbb372; no voters: 510a02d078ff4965b5dfee7317012028, c34083cb0fd441ec933ad27e4602ee5e
I20260928 15:00:04.241681 15003 raft_consensus.cc:2726] T 40875a7e94c840fdbfa8e5dc020db5b3 P 99586f213ec449ef8f1a8b1255dbb372 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 15:00:04.316402 15004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:04.315460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.2:49705)
0928 15:00:04.316059 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:04.316166 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:43875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":299,"threads_started":1}
I20260928 15:00:04.754252 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 14522
I20260928 15:00:04.777088 15005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:04.755426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58650 (local address 127.31.40.1:45251)
0928 15:00:04.756137 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:04.756144 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:04.756171 (+ 27us) server_negotiation.cc:407] Connection header received
0928 15:00:04.756236 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:04.756242 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:04.756592 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:04.756771 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:00:04.758938 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:04.760003 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:04.761355 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:04.762011 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:04.762121 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:04.771114 (+ 8993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:00:04.771131 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:00:04.771149 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:00:04.771191 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:00:04.774166 (+ 2975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:04.774934 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:04.774955 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:04.774958 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:04.775131 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:04.775531 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:04.775535 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:04.775537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:04.776042 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:00:04.776306 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:04.776572 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 15:00:04.776781 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":345,"threads_started":1}
I20260928 15:00:04.778638 14681 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:00:04.778941 15006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:04.778147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:43875 (local address 127.31.40.3:53841)
0928 15:00:04.778617 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:04.778767 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:43875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":238,"threads_started":1}
I20260928 15:00:04.872804 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 14522
I20260928 15:00:04.892071 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 14690
I20260928 15:00:04.905754 15009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:04.892825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40746 (local address 127.31.40.2:38411)
0928 15:00:04.893317 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:04.893322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:04.893339 (+ 17us) server_negotiation.cc:407] Connection header received
0928 15:00:04.893401 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:04.893405 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:04.893467 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:04.893559 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:00:04.894866 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:04.895627 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:04.896945 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:04.897344 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:04.897444 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:04.900185 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:00:04.900192 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:00:04.900214 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:00:04.900259 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:00:04.903423 (+ 3164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:04.904128 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:04.904132 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:04.904135 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:04.904274 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:04.904594 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:04.904597 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:04.904599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:04.904869 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:00:04.905069 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:04.905291 (+ 222us) server_negotiation.cc:299] Negotiation successful
0928 15:00:04.905473 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":279,"threads_started":1}
I20260928 15:00:04.906508 14813 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:00:04.992835 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 14690
I20260928 15:00:05.011353 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 14821
I20260928 15:00:05.025043 15011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:05.012267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37908 (local address 127.31.40.3:39831)
0928 15:00:05.012858 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:05.012862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:05.012883 (+ 21us) server_negotiation.cc:407] Connection header received
0928 15:00:05.012941 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:05.012945 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:05.013014 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:05.013106 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:00:05.014480 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:05.015296 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:05.016724 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:05.017158 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:05.017255 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:05.019917 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:00:05.019926 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:00:05.019934 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:00:05.019967 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:00:05.022571 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:05.023239 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:05.023242 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:05.023244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:05.023402 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:05.023730 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:05.023733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:05.023735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:05.024116 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:00:05.024341 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:05.024590 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 15:00:05.024751 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":326,"threads_started":1}
I20260928 15:00:05.025753 14944 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:00:05.110838 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 14821
2026-09-28T15:00:05Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24826 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:00:05 dist-test-slave-j968 krb5kdc[15015](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:00:05 dist-test-slave-j968 krb5kdc[15015](info): set up 2 sockets
Sep 28 15:00:05 dist-test-slave-j968 krb5kdc[15015](info): commencing operation
krb5kdc: starting...
W20260928 15:00:06.225648 31904 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Sep 28 15:00:06 dist-test-slave-j968 krb5kdc[15015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:00:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:00:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab.
Entry for principal kudu/127.31.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab.
I20260928 15:00:06.483157 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:34035
--webserver_interface=127.31.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:34399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.40.62:34035
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:00:06.835572 15028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:00:06.836012 15028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:00:06.836220 15028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:00:06.844692 15028 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 15:00:06.844856 15028 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 15:00:06.844933 15028 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 15:00:06.845024 15028 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:00:06.845103 15028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:00:06.845208 15028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:00:06.845299 15028 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:00:06.845378 15028 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:00:06.855643 15028 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:34399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.31.40.62:34035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:34035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:00:06.857155 15028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:00:06.858565 15028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:00:06.871896 15033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:06.871851 15036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:06.871970 15034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:00:06.872181 15028 server_base.cc:1034] running on GCE node
I20260928 15:00:06.873369 15028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:00:06.874873 15028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:00:06.876093 15028 hybrid_clock.cc:648] HybridClock initialized: now 1790607606876031 us; error 27 us; skew 500 ppm
I20260928 15:00:06.876475 15028 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:00:06 dist-test-slave-j968 krb5kdc[15015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607606, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:00:06.879758 15028 init.cc:377] Logged in from keytab as kudu/127.31.40.62@KRBTEST.COM (short username kudu)
I20260928 15:00:06.881253 15028 webserver.cc:460] Webserver started at http://127.31.40.62:36777/ using document root <none> and password file <none>
I20260928 15:00:06.881979 15028 fs_manager.cc:359] Metadata directory not provided
I20260928 15:00:06.882076 15028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:00:06.882349 15028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:00:06.885371 15028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/master-0/data/instance:
uuid: "99120398a8b0443aa9313e4874f4aa69"
format_stamp: "Formatted at 2026-09-28 15:00:06 on dist-test-slave-j968"
server_key: "c032cd757ec4b7ee1e8e39128856c2ea"
server_key_iv: "84998d5837221dcbb3e30753c62ad036"
server_key_version: "encryptionkey@0"
I20260928 15:00:06.886127 15028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/master-0/wal/instance:
uuid: "99120398a8b0443aa9313e4874f4aa69"
format_stamp: "Formatted at 2026-09-28 15:00:06 on dist-test-slave-j968"
server_key: "c032cd757ec4b7ee1e8e39128856c2ea"
server_key_iv: "84998d5837221dcbb3e30753c62ad036"
server_key_version: "encryptionkey@0"
I20260928 15:00:06.891467 15028 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260928 15:00:06.895015 15043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:06.896641 15028 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:00:06.896853 15028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/master-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/master-0/wal
uuid: "99120398a8b0443aa9313e4874f4aa69"
format_stamp: "Formatted at 2026-09-28 15:00:06 on dist-test-slave-j968"
server_key: "c032cd757ec4b7ee1e8e39128856c2ea"
server_key_iv: "84998d5837221dcbb3e30753c62ad036"
server_key_version: "encryptionkey@0"
I20260928 15:00:06.897109 15028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:00:06.932065 15028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:00:06.937000 15028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:00:06.937295 15028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:00:06.956869 15028 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.62:34035
I20260928 15:00:06.956887 15094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.62:34035 every 8 connection(s)
I20260928 15:00:06.958333 15028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
I20260928 15:00:06.962416 15095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:00:06.968837 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 15028
I20260928 15:00:06.969085 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/master-0/wal/instance
I20260928 15:00:06.969960 31904 external_mini_cluster.cc:1428] Setting key ea18e75f54ee9dc434a41338a27ce8c0
I20260928 15:00:06.974038 15095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69: Bootstrap starting.
I20260928 15:00:06.977900 15095 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69: Neither blocks nor log segments found. Creating new log.
I20260928 15:00:06.979007 15095 log.cc:824] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69: Log is configured to *not* fsync() on all Append() calls
I20260928 15:00:06.982333 15095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69: No bootstrap required, opened a new log
Sep 28 15:00:06 dist-test-slave-j968 krb5kdc[15015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 15:00:06.991109 15095 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99120398a8b0443aa9313e4874f4aa69" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34035 } }
I20260928 15:00:06.991549 15095 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:00:06.991662 15095 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99120398a8b0443aa9313e4874f4aa69, State: Initialized, Role: FOLLOWER
I20260928 15:00:06.992471 15095 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99120398a8b0443aa9313e4874f4aa69" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34035 } }
I20260928 15:00:06.992779 15095 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:00:06.992939 15095 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:00:06.993140 15095 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:00:06.995004 15098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:06.972282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47832 (local address 127.31.40.62:34035)
0928 15:00:06.973134 (+ 852us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:06.973147 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:06.973195 (+ 48us) server_negotiation.cc:407] Connection header received
0928 15:00:06.974992 (+ 1797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:06.975016 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:06.975432 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:06.975779 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:00:06.977718 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:06.979262 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:06.980961 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:06.981638 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:06.981796 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:06.986854 (+ 5058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:00:06.986867 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:00:06.986883 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:00:06.986924 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:00:06.990282 (+ 3358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:06.991072 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:06.991079 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:06.991086 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:06.991260 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:06.991792 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:06.991797 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:06.991800 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:06.992605 (+ 805us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:00:06.992909 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:06.993432 (+ 523us) server_negotiation.cc:299] Negotiation successful
0928 15:00:06.993906 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":404,"threads_started":1}
I20260928 15:00:06.995776 15095 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99120398a8b0443aa9313e4874f4aa69" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34035 } }
I20260928 15:00:06.996395 15095 leader_election.cc:302] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99120398a8b0443aa9313e4874f4aa69; no voters:
I20260928 15:00:06.997105 15095 leader_election.cc:288] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:00:06.997210 15100 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:00:06.998401 15100 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69 [term 1 LEADER]: Becoming Leader. State: Replica: 99120398a8b0443aa9313e4874f4aa69, State: Running, Role: LEADER
I20260928 15:00:06.999032 15100 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99120398a8b0443aa9313e4874f4aa69" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34035 } }
I20260928 15:00:06.999575 15095 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:00:07.002218 15101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99120398a8b0443aa9313e4874f4aa69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99120398a8b0443aa9313e4874f4aa69" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34035 } } }
I20260928 15:00:07.002624 15101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69 [sys.catalog]: This master's current role is: LEADER
I20260928 15:00:07.003477 15102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99120398a8b0443aa9313e4874f4aa69. Latest consensus state: current_term: 1 leader_uuid: "99120398a8b0443aa9313e4874f4aa69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99120398a8b0443aa9313e4874f4aa69" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34035 } } }
I20260928 15:00:07.003938 15102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69 [sys.catalog]: This master's current role is: LEADER
I20260928 15:00:07.005141 15106 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:00:07.011598 15106 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:00:07.021876 15106 catalog_manager.cc:1347] Generated new cluster ID: bfc018e2a8b7437587bf5b8abf7d1632
I20260928 15:00:07.021997 15106 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:00:07.066087 15106 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:00:07.068049 15106 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:00:07.080106 15106 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69: Generated new TSK 0
I20260928 15:00:07.081127 15106 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.40.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab.
Entry for principal kudu/127.31.40.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab.
I20260928 15:00:07.148240 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.1:0
--local_ip_for_outbound_sockets=127.31.40.1
--webserver_interface=127.31.40.1
--webserver_port=0
--tserver_master_addrs=127.31.40.62:34035
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab
--principal=kudu/127.31.40.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:34399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:00:07.490231 15121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:00:07.490512 15121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:00:07.490605 15121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:00:07.498169 15121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:00:07.498289 15121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:00:07.498404 15121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.1
I20260928 15:00:07.508425 15121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:34399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab
--principal=kudu/127.31.40.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.40.1
--webserver_port=0
--tserver_master_addrs=127.31.40.62:34035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.1
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:00:07.509752 15121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:00:07.510995 15121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:00:07.524250 15126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:07.524174 15127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:07.524204 15129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:00:07.524996 15121 server_base.cc:1034] running on GCE node
I20260928 15:00:07.525627 15121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:00:07.526640 15121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:00:07.527945 15121 hybrid_clock.cc:648] HybridClock initialized: now 1790607607527837 us; error 47 us; skew 500 ppm
I20260928 15:00:07.528290 15121 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:00:07 dist-test-slave-j968 krb5kdc[15015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607607, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:00:07.531462 15121 init.cc:377] Logged in from keytab as kudu/127.31.40.1@KRBTEST.COM (short username kudu)
I20260928 15:00:07.532912 15121 webserver.cc:460] Webserver started at http://127.31.40.1:38983/ using document root <none> and password file <none>
I20260928 15:00:07.533603 15121 fs_manager.cc:359] Metadata directory not provided
I20260928 15:00:07.533756 15121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:00:07.534070 15121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:00:07.537047 15121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-0/data/instance:
uuid: "14b53f4d618d4bfb96ce8b6fc0746dc0"
format_stamp: "Formatted at 2026-09-28 15:00:07 on dist-test-slave-j968"
server_key: "ee4d4c5d194c7304fecefb3eef4fd5e2"
server_key_iv: "f6b65b0ead875215fb8c94b737eab460"
server_key_version: "encryptionkey@0"
I20260928 15:00:07.537797 15121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-0/wal/instance:
uuid: "14b53f4d618d4bfb96ce8b6fc0746dc0"
format_stamp: "Formatted at 2026-09-28 15:00:07 on dist-test-slave-j968"
server_key: "ee4d4c5d194c7304fecefb3eef4fd5e2"
server_key_iv: "f6b65b0ead875215fb8c94b737eab460"
server_key_version: "encryptionkey@0"
I20260928 15:00:07.542992 15121 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 15:00:07.546557 15136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:07.548072 15121 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:00:07.548285 15121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-0/wal
uuid: "14b53f4d618d4bfb96ce8b6fc0746dc0"
format_stamp: "Formatted at 2026-09-28 15:00:07 on dist-test-slave-j968"
server_key: "ee4d4c5d194c7304fecefb3eef4fd5e2"
server_key_iv: "f6b65b0ead875215fb8c94b737eab460"
server_key_version: "encryptionkey@0"
I20260928 15:00:07.548506 15121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:00:07.576085 15121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:00:07.580888 15121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:00:07.581190 15121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:00:07.582196 15121 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:00:07.583694 15121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:00:07.583806 15121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:07.583974 15121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:00:07.584048 15121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:07.618008 15121 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.1:32837
I20260928 15:00:07.618039 15248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.1:32837 every 8 connection(s)
I20260928 15:00:07.619397 15121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
I20260928 15:00:07.626844 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 15121
I20260928 15:00:07.627084 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-0/wal/instance
I20260928 15:00:07.627851 31904 external_mini_cluster.cc:1428] Setting key c46766773366592ed4e4d114c565ffc8
Sep 28 15:00:07 dist-test-slave-j968 krb5kdc[15015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607607, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.1@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 15:00:07.642603 15251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:07.622101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:33621 (local address 127.31.40.62:34035)
0928 15:00:07.622566 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:07.622570 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:07.623490 (+ 920us) server_negotiation.cc:407] Connection header received
0928 15:00:07.625458 (+ 1968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:07.625463 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:07.625555 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:07.625674 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:00:07.627336 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:07.628190 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:07.629960 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:07.630627 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:07.630791 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:07.634972 (+ 4181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:00:07.634982 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:00:07.634990 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:00:07.635039 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:00:07.638941 (+ 3902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:07.640081 (+ 1140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:07.640086 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:07.640088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:07.640215 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:07.640757 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:07.640761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:07.640763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:07.641041 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:00:07.641155 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:07.642116 (+ 961us) server_negotiation.cc:299] Negotiation successful
0928 15:00:07.642300 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":248,"threads_started":1}
I20260928 15:00:07.644235 15252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:07.622574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34035 (local address 127.31.40.1:33621)
0928 15:00:07.623301 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:07.623364 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:07.625156 (+ 1792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:07.625917 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:07.625926 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:07.626465 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:00:07.627122 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:07.627165 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:07.628361 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:07.628366 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:07.629800 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:07.629816 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:07.630900 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:07.630907 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:07.631123 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:07.632028 (+ 905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:00:07.632052 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:00:07.634706 (+ 2654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:00:07.639149 (+ 4443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:07.639162 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:07.639186 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:07.639946 (+ 760us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:07.640341 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:07.640348 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:07.640352 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:07.640625 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:07.641171 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:00:07.641183 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:00:07.641674 (+ 491us) client_negotiation.cc:769] Sending connection context
0928 15:00:07.642013 (+ 339us) client_negotiation.cc:240] Negotiation successful
0928 15:00:07.642630 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":303,"threads_started":1}
I20260928 15:00:07.645972 15249 heartbeater.cc:347] Connected to a master server at 127.31.40.62:34035
I20260928 15:00:07.646302 15249 heartbeater.cc:464] Registering TS with master...
I20260928 15:00:07.647092 15249 heartbeater.cc:511] Master 127.31.40.62:34035 requested a full tablet report, sending...
I20260928 15:00:07.649252 15060 ts_manager.cc:194] Registered new tserver with Master: 14b53f4d618d4bfb96ce8b6fc0746dc0 (127.31.40.1:32837)
I20260928 15:00:07.650743 15060 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.1@KRBTEST.COM'} at 127.31.40.1:33621
WARNING: no policy specified for kudu/127.31.40.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab.
Entry for principal kudu/127.31.40.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab.
I20260928 15:00:07.686555 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.2:0
--local_ip_for_outbound_sockets=127.31.40.2
--webserver_interface=127.31.40.2
--webserver_port=0
--tserver_master_addrs=127.31.40.62:34035
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab
--principal=kudu/127.31.40.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:34399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:00:08.024447 15255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:00:08.024729 15255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:00:08.024823 15255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:00:08.032034 15255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:00:08.032140 15255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:00:08.032253 15255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.2
I20260928 15:00:08.042160 15255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:34399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab
--principal=kudu/127.31.40.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.40.2
--webserver_port=0
--tserver_master_addrs=127.31.40.62:34035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.2
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:00:08.043455 15255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:00:08.044809 15255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:00:08.059044 15263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:08.059145 15260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:08.059145 15261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:00:08.060153 15255 server_base.cc:1034] running on GCE node
I20260928 15:00:08.060853 15255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:00:08.061959 15255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:00:08.063233 15255 hybrid_clock.cc:648] HybridClock initialized: now 1790607608063147 us; error 74 us; skew 500 ppm
I20260928 15:00:08.063609 15255 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:00:08 dist-test-slave-j968 krb5kdc[15015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607608, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:00:08.067034 15255 init.cc:377] Logged in from keytab as kudu/127.31.40.2@KRBTEST.COM (short username kudu)
I20260928 15:00:08.068706 15255 webserver.cc:460] Webserver started at http://127.31.40.2:36365/ using document root <none> and password file <none>
I20260928 15:00:08.069478 15255 fs_manager.cc:359] Metadata directory not provided
I20260928 15:00:08.069628 15255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:00:08.069953 15255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:00:08.073027 15255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-1/data/instance:
uuid: "3559026efbc642419cabdda2898714a6"
format_stamp: "Formatted at 2026-09-28 15:00:08 on dist-test-slave-j968"
server_key: "bcb92fd637f53804a8d6c6b0460b1226"
server_key_iv: "3c0fd12e9261b1e47506f8167fe68de4"
server_key_version: "encryptionkey@0"
I20260928 15:00:08.073856 15255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-1/wal/instance:
uuid: "3559026efbc642419cabdda2898714a6"
format_stamp: "Formatted at 2026-09-28 15:00:08 on dist-test-slave-j968"
server_key: "bcb92fd637f53804a8d6c6b0460b1226"
server_key_iv: "3c0fd12e9261b1e47506f8167fe68de4"
server_key_version: "encryptionkey@0"
I20260928 15:00:08.079267 15255 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 15:00:08.083002 15270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:08.084450 15255 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:00:08.084632 15255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-1/wal
uuid: "3559026efbc642419cabdda2898714a6"
format_stamp: "Formatted at 2026-09-28 15:00:08 on dist-test-slave-j968"
server_key: "bcb92fd637f53804a8d6c6b0460b1226"
server_key_iv: "3c0fd12e9261b1e47506f8167fe68de4"
server_key_version: "encryptionkey@0"
I20260928 15:00:08.084851 15255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:00:08.101205 15255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:00:08.105633 15255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:00:08.105953 15255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:00:08.106948 15255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:00:08.108405 15255 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:00:08.108512 15255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:08.108676 15255 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:00:08.108795 15255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260928 15:00:08.157569 15255 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.2:32855
I20260928 15:00:08.157615 15382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.2:32855 every 8 connection(s)
I20260928 15:00:08.159015 15255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
I20260928 15:00:08.162328 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 15255
I20260928 15:00:08.162539 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-1/wal/instance
I20260928 15:00:08.163110 31904 external_mini_cluster.cc:1428] Setting key 969305fc1ddf122e82fcec9a6c21380c
Sep 28 15:00:08 dist-test-slave-j968 krb5kdc[15015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607608, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.2@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 15:00:08.182505 15385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:08.162037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:32975 (local address 127.31.40.62:34035)
0928 15:00:08.162609 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:08.162615 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:08.163929 (+ 1314us) server_negotiation.cc:407] Connection header received
0928 15:00:08.166137 (+ 2208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:08.166145 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:08.166234 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:08.166359 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:00:08.168139 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:08.169303 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:08.171126 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:08.171906 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:08.172047 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:08.176484 (+ 4437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:00:08.176489 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:00:08.176493 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:00:08.176529 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:00:08.179245 (+ 2716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:08.180090 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:08.180095 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:08.180097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:08.180226 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:08.180694 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:08.180703 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:08.180705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:08.180987 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:00:08.181151 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:08.181884 (+ 733us) server_negotiation.cc:299] Negotiation successful
0928 15:00:08.182114 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":283,"threads_started":1}
I20260928 15:00:08.183605 15386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:08.162626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34035 (local address 127.31.40.2:32975)
0928 15:00:08.163668 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:08.163717 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:08.165843 (+ 2126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:08.166578 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:08.166586 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:08.167241 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:00:08.167956 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:08.167979 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:08.169493 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:08.169497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:08.170968 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:08.170980 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:08.172132 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:08.172138 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:08.172378 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:08.173228 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:00:08.173264 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:00:08.176223 (+ 2959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:00:08.179403 (+ 3180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:08.179413 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:08.179433 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:08.179949 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:08.180344 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:08.180350 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:08.180353 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:08.180568 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:08.181122 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:00:08.181131 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:00:08.181507 (+ 376us) client_negotiation.cc:769] Sending connection context
0928 15:00:08.181796 (+ 289us) client_negotiation.cc:240] Negotiation successful
0928 15:00:08.182382 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"thread_start_us":504,"threads_started":1}
I20260928 15:00:08.185160 15383 heartbeater.cc:347] Connected to a master server at 127.31.40.62:34035
I20260928 15:00:08.185541 15383 heartbeater.cc:464] Registering TS with master...
I20260928 15:00:08.186568 15383 heartbeater.cc:511] Master 127.31.40.62:34035 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.31.40.3@KRBTEST.COM; defaulting to no policy
I20260928 15:00:08.188441 15060 ts_manager.cc:194] Registered new tserver with Master: 3559026efbc642419cabdda2898714a6 (127.31.40.2:32855)
I20260928 15:00:08.189393 15060 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.2@KRBTEST.COM'} at 127.31.40.2:32975
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab.
Entry for principal kudu/127.31.40.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab.
I20260928 15:00:08.219771 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.3:0
--local_ip_for_outbound_sockets=127.31.40.3
--webserver_interface=127.31.40.3
--webserver_port=0
--tserver_master_addrs=127.31.40.62:34035
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab
--principal=kudu/127.31.40.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:34399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:00:08.563074 15389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:00:08.563452 15389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:00:08.563648 15389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:00:08.572304 15389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:00:08.572479 15389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:00:08.572701 15389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.3
I20260928 15:00:08.583309 15389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:34399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab
--principal=kudu/127.31.40.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.40.3
--webserver_port=0
--tserver_master_addrs=127.31.40.62:34035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.3
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:00:08.584774 15389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:00:08.586110 15389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:00:08.600355 15394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:08.600585 15395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:08.600883 15397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:00:08.602016 15389 server_base.cc:1034] running on GCE node
I20260928 15:00:08.602595 15389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:00:08.603614 15389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:00:08.604817 15389 hybrid_clock.cc:648] HybridClock initialized: now 1790607608604751 us; error 61 us; skew 500 ppm
I20260928 15:00:08.605110 15389 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:00:08 dist-test-slave-j968 krb5kdc[15015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607608, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:00:08.608387 15389 init.cc:377] Logged in from keytab as kudu/127.31.40.3@KRBTEST.COM (short username kudu)
I20260928 15:00:08.609954 15389 webserver.cc:460] Webserver started at http://127.31.40.3:33821/ using document root <none> and password file <none>
I20260928 15:00:08.610677 15389 fs_manager.cc:359] Metadata directory not provided
I20260928 15:00:08.610816 15389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:00:08.611130 15389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:00:08.614157 15389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-2/data/instance:
uuid: "e3b9cbb8fe0945beb891b825281d5bfb"
format_stamp: "Formatted at 2026-09-28 15:00:08 on dist-test-slave-j968"
server_key: "3190af6063ecb0bbbd71d9775df3a410"
server_key_iv: "1e56583a12a5b4b0eb45352f18a991b5"
server_key_version: "encryptionkey@0"
I20260928 15:00:08.614970 15389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-2/wal/instance:
uuid: "e3b9cbb8fe0945beb891b825281d5bfb"
format_stamp: "Formatted at 2026-09-28 15:00:08 on dist-test-slave-j968"
server_key: "3190af6063ecb0bbbd71d9775df3a410"
server_key_iv: "1e56583a12a5b4b0eb45352f18a991b5"
server_key_version: "encryptionkey@0"
I20260928 15:00:08.620646 15389 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 15:00:08.624462 15404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:08.625902 15389 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 15:00:08.626084 15389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-2/wal
uuid: "e3b9cbb8fe0945beb891b825281d5bfb"
format_stamp: "Formatted at 2026-09-28 15:00:08 on dist-test-slave-j968"
server_key: "3190af6063ecb0bbbd71d9775df3a410"
server_key_iv: "1e56583a12a5b4b0eb45352f18a991b5"
server_key_version: "encryptionkey@0"
I20260928 15:00:08.626273 15389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:00:08.643105 15389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:00:08.647941 15389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:00:08.648248 15389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:00:08.649274 15389 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:00:08.650664 15389 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:00:08.650738 15389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:08.650924 15389 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:00:08.650976 15389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:08.654346 15249 heartbeater.cc:503] Master 127.31.40.62:34035 was elected leader, sending a full tablet report...
I20260928 15:00:08.687290 15389 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.3:46849
I20260928 15:00:08.687337 15516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.3:46849 every 8 connection(s)
I20260928 15:00:08.688786 15389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
I20260928 15:00:08.695302 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 15389
I20260928 15:00:08.695547 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-2/wal/instance
I20260928 15:00:08.696110 31904 external_mini_cluster.cc:1428] Setting key 1bba854a49c69a91975bf35d77d98e3a
Sep 28 15:00:08 dist-test-slave-j968 krb5kdc[15015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607608, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.3@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 15:00:08.711481 15519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:08.691930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:48911 (local address 127.31.40.62:34035)
0928 15:00:08.692458 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:08.692466 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:08.693561 (+ 1095us) server_negotiation.cc:407] Connection header received
0928 15:00:08.695392 (+ 1831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:08.695398 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:08.695498 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:08.695619 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:00:08.697508 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:08.698295 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:08.699934 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:08.700413 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:08.700545 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:08.704726 (+ 4181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:00:08.704731 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:00:08.704734 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:00:08.704827 (+ 93us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:00:08.707489 (+ 2662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:08.708451 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:08.708456 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:08.708458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:08.708590 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:08.709120 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:08.709123 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:08.709125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:08.709555 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:00:08.709692 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:08.710896 (+ 1204us) server_negotiation.cc:299] Negotiation successful
0928 15:00:08.711105 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":210,"threads_started":1}
I20260928 15:00:08.712806 15520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:08.692555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34035 (local address 127.31.40.3:48911)
0928 15:00:08.693383 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:08.693429 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:08.695088 (+ 1659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:08.695931 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:08.695949 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:08.696564 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:00:08.697324 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:08.697342 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:08.698468 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:08.698472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:08.699741 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:08.699767 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:08.700633 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:08.700640 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:08.700975 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:08.701943 (+ 968us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:00:08.701983 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:00:08.704510 (+ 2527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:00:08.707646 (+ 3136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:08.707654 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:08.707669 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:08.708298 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:08.708702 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:08.708706 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:08.708708 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:08.708963 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:08.709824 (+ 861us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:00:08.709837 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:00:08.710372 (+ 535us) client_negotiation.cc:769] Sending connection context
0928 15:00:08.710722 (+ 350us) client_negotiation.cc:240] Negotiation successful
0928 15:00:08.711382 (+ 660us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":318,"threads_started":1}
I20260928 15:00:08.714283 15517 heartbeater.cc:347] Connected to a master server at 127.31.40.62:34035
I20260928 15:00:08.714634 15517 heartbeater.cc:464] Registering TS with master...
I20260928 15:00:08.715505 15517 heartbeater.cc:511] Master 127.31.40.62:34035 requested a full tablet report, sending...
I20260928 15:00:08.717027 15060 ts_manager.cc:194] Registered new tserver with Master: e3b9cbb8fe0945beb891b825281d5bfb (127.31.40.3:46849)
I20260928 15:00:08.717813 15060 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.3@KRBTEST.COM'} at 127.31.40.3:48911
I20260928 15:00:08.724556 31904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:00:08.743582 15519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:08.729907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45130 (local address 127.31.40.62:34035)
0928 15:00:08.730103 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:08.730109 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:08.730625 (+ 516us) server_negotiation.cc:407] Connection header received
0928 15:00:08.730821 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:08.730826 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:08.730922 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:08.731013 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:00:08.732874 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:08.733656 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:08.735171 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:08.735609 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:08.735711 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:08.738286 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:00:08.738291 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:00:08.738294 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:00:08.738338 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:00:08.741034 (+ 2696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:08.741692 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:08.741696 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:08.741698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:08.741845 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:08.742198 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:08.742201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:08.742203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:08.742548 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:00:08.742742 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:08.743127 (+ 385us) server_negotiation.cc:299] Negotiation successful
0928 15:00:08.743306 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 15:00:08.748016 15060 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45130:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:00:08.755198 15060 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:00:08.780741 15530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:08.768517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.3:46849 (local address 127.0.0.1:43006)
0928 15:00:08.769323 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:08.769383 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:08.769565 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:08.771596 (+ 2031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:08.771604 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:08.771642 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:00:08.772140 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:08.772155 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:08.774779 (+ 2624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:08.774785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:08.777734 (+ 2949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:08.777750 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:08.779516 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:08.779530 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:08.779754 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:08.779833 (+ 79us) client_negotiation.cc:769] Sending connection context
0928 15:00:08.779998 (+ 165us) client_negotiation.cc:240] Negotiation successful
0928 15:00:08.780196 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"mutex_wait_us":25,"thread_start_us":377,"threads_started":1}
I20260928 15:00:08.781535 15532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:08.768911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43006 (local address 127.31.40.3:46849)
0928 15:00:08.770864 (+ 1953us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:08.770872 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:08.770909 (+ 37us) server_negotiation.cc:407] Connection header received
0928 15:00:08.771038 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:08.771047 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:08.771347 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:08.772399 (+ 1052us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:00:08.772707 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:08.774600 (+ 1893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:08.777930 (+ 3330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:08.779313 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:08.779450 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:08.779701 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:08.780987 (+ 1286us) server_negotiation.cc:299] Negotiation successful
0928 15:00:08.781155 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1757,"thread_start_us":1555,"threads_started":1}
I20260928 15:00:08.781904 15533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:08.769788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57750 (local address 127.31.40.2:32855)
0928 15:00:08.770400 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:08.770413 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:08.771076 (+ 663us) server_negotiation.cc:407] Connection header received
0928 15:00:08.771384 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:08.771395 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:08.771655 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:08.771857 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:00:08.772944 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:08.774600 (+ 1656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:08.778721 (+ 4121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:08.780225 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:08.780344 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:08.780531 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:08.781312 (+ 781us) server_negotiation.cc:299] Negotiation successful
0928 15:00:08.781521 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":320,"threads_started":1}
I20260928 15:00:08.782208 15534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:08.768602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.2:32855 (local address 127.0.0.1:57750)
0928 15:00:08.770944 (+ 2342us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:08.770970 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:08.771182 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:08.772190 (+ 1008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:08.772197 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:08.772231 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:00:08.772720 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:08.772736 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:08.774792 (+ 2056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:08.774796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:08.778522 (+ 3726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:08.778541 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:08.780705 (+ 2164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:08.780712 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:08.781095 (+ 383us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:08.781130 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 15:00:08.781222 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 15:00:08.781371 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2201,"mutex_wait_us":144,"thread_start_us":8337,"threads_started":1}
I20260928 15:00:08.785516 15452 tablet_service.cc:1467] Processing CreateTablet for tablet 2bdcdcf5f8d14c608d7d9d445f993b26 (DEFAULT_TABLE table=test-table [id=5acfdf07a48c4dbd9257bb468b95b7d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:00:08.786631 15531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:08.770338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:32837 (local address 127.0.0.1:44962)
0928 15:00:08.777239 (+ 6901us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:08.777268 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:08.777399 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:08.778095 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:08.778100 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:08.778129 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:00:08.778577 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:08.778592 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:08.780646 (+ 2054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:08.780652 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:08.783733 (+ 3081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:08.783753 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:08.785861 (+ 2108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:08.785867 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:08.786051 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:08.786082 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 15:00:08.786183 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 15:00:08.786318 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6758,"thread_start_us":861,"threads_started":1}
I20260928 15:00:08.786774 15535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:08.771010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44962 (local address 127.31.40.1:32837)
0928 15:00:08.776798 (+ 5788us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:08.776807 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:08.777541 (+ 734us) server_negotiation.cc:407] Connection header received
0928 15:00:08.777639 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:08.777645 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:08.777883 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:08.778091 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:00:08.778760 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:08.780460 (+ 1700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:08.784055 (+ 3595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:08.785646 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:08.785792 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:08.786035 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:08.786245 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 15:00:08.786429 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5593,"thread_start_us":5428,"threads_started":1}
I20260928 15:00:08.786437 15318 tablet_service.cc:1467] Processing CreateTablet for tablet 2bdcdcf5f8d14c608d7d9d445f993b26 (DEFAULT_TABLE table=test-table [id=5acfdf07a48c4dbd9257bb468b95b7d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:00:08.787262 15452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bdcdcf5f8d14c608d7d9d445f993b26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:00:08.787962 15318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bdcdcf5f8d14c608d7d9d445f993b26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:00:08.790717 15184 tablet_service.cc:1467] Processing CreateTablet for tablet 2bdcdcf5f8d14c608d7d9d445f993b26 (DEFAULT_TABLE table=test-table [id=5acfdf07a48c4dbd9257bb468b95b7d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:00:08.792191 15184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bdcdcf5f8d14c608d7d9d445f993b26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:00:08.799770 15536 tablet_bootstrap.cc:492] T 2bdcdcf5f8d14c608d7d9d445f993b26 P e3b9cbb8fe0945beb891b825281d5bfb: Bootstrap starting.
I20260928 15:00:08.802937 15536 tablet_bootstrap.cc:654] T 2bdcdcf5f8d14c608d7d9d445f993b26 P e3b9cbb8fe0945beb891b825281d5bfb: Neither blocks nor log segments found. Creating new log.
I20260928 15:00:08.804134 15536 log.cc:824] T 2bdcdcf5f8d14c608d7d9d445f993b26 P e3b9cbb8fe0945beb891b825281d5bfb: Log is configured to *not* fsync() on all Append() calls
I20260928 15:00:08.804653 15537 tablet_bootstrap.cc:492] T 2bdcdcf5f8d14c608d7d9d445f993b26 P 3559026efbc642419cabdda2898714a6: Bootstrap starting.
I20260928 15:00:08.808527 15536 tablet_bootstrap.cc:492] T 2bdcdcf5f8d14c608d7d9d445f993b26 P e3b9cbb8fe0945beb891b825281d5bfb: No bootstrap required, opened a new log
I20260928 15:00:08.808547 15537 tablet_bootstrap.cc:654] T 2bdcdcf5f8d14c608d7d9d445f993b26 P 3559026efbc642419cabdda2898714a6: Neither blocks nor log segments found. Creating new log.
I20260928 15:00:08.808974 15536 ts_tablet_manager.cc:1397] T 2bdcdcf5f8d14c608d7d9d445f993b26 P e3b9cbb8fe0945beb891b825281d5bfb: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20260928 15:00:08.809602 15537 log.cc:824] T 2bdcdcf5f8d14c608d7d9d445f993b26 P 3559026efbc642419cabdda2898714a6: Log is configured to *not* fsync() on all Append() calls
I20260928 15:00:08.810170 15539 tablet_bootstrap.cc:492] T 2bdcdcf5f8d14c608d7d9d445f993b26 P 14b53f4d618d4bfb96ce8b6fc0746dc0: Bootstrap starting.
I20260928 15:00:08.813978 15539 tablet_bootstrap.cc:654] T 2bdcdcf5f8d14c608d7d9d445f993b26 P 14b53f4d618d4bfb96ce8b6fc0746dc0: Neither blocks nor log segments found. Creating new log.
I20260928 15:00:08.814795 15537 tablet_bootstrap.cc:492] T 2bdcdcf5f8d14c608d7d9d445f993b26 P 3559026efbc642419cabdda2898714a6: No bootstrap required, opened a new log
I20260928 15:00:08.815141 15537 ts_tablet_manager.cc:1397] T 2bdcdcf5f8d14c608d7d9d445f993b26 P 3559026efbc642419cabdda2898714a6: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260928 15:00:08.815271 15539 log.cc:824] T 2bdcdcf5f8d14c608d7d9d445f993b26 P 14b53f4d618d4bfb96ce8b6fc0746dc0: Log is configured to *not* fsync() on all Append() calls
I20260928 15:00:08.818591 15539 tablet_bootstrap.cc:492] T 2bdcdcf5f8d14c608d7d9d445f993b26 P 14b53f4d618d4bfb96ce8b6fc0746dc0: No bootstrap required, opened a new log
I20260928 15:00:08.818862 15539 ts_tablet_manager.cc:1397] T 2bdcdcf5f8d14c608d7d9d445f993b26 P 14b53f4d618d4bfb96ce8b6fc0746dc0: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260928 15:00:08.818852 15536 raft_consensus.cc:348] T 2bdcdcf5f8d14c608d7d9d445f993b26 P e3b9cbb8fe0945beb891b825281d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b9cbb8fe0945beb891b825281d5bfb" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 46849 } } peers { permanent_uuid: "3559026efbc642419cabdda2898714a6" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 32855 } } peers { permanent_uuid: "14b53f4d618d4bfb96ce8b6fc0746dc0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 32837 } }
I20260928 15:00:08.819312 15536 raft_consensus.cc:374] T 2bdcdcf5f8d14c608d7d9d445f993b26 P e3b9cbb8fe0945beb891b825281d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:00:08.819451 15536 raft_consensus.cc:729] T 2bdcdcf5f8d14c608d7d9d445f993b26 P e3b9cbb8fe0945beb891b825281d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3b9cbb8fe0945beb891b825281d5bfb, State: Initialized, Role: FOLLOWER
I20260928 15:00:08.820143 15536 consensus_queue.cc:260] T 2bdcdcf5f8d14c608d7d9d445f993b26 P e3b9cbb8fe0945beb891b825281d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b9cbb8fe0945beb891b825281d5bfb" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 46849 } } peers { permanent_uuid: "3559026efbc642419cabdda2898714a6" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 32855 } } peers { permanent_uuid: "14b53f4d618d4bfb96ce8b6fc0746dc0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 32837 } }
I20260928 15:00:08.821300 15517 heartbeater.cc:503] Master 127.31.40.62:34035 was elected leader, sending a full tablet report...
I20260928 15:00:08.823338 15536 ts_tablet_manager.cc:1428] T 2bdcdcf5f8d14c608d7d9d445f993b26 P e3b9cbb8fe0945beb891b825281d5bfb: Time spent starting tablet: real 0.014s user 0.007s sys 0.005s
I20260928 15:00:08.825910 15537 raft_consensus.cc:348] T 2bdcdcf5f8d14c608d7d9d445f993b26 P 3559026efbc642419cabdda2898714a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b9cbb8fe0945beb891b825281d5bfb" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 46849 } } peers { permanent_uuid: "3559026efbc642419cabdda2898714a6" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 32855 } } peers { permanent_uuid: "14b53f4d618d4bfb96ce8b6fc0746dc0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 32837 } }
I20260928 15:00:08.826503 15537 raft_consensus.cc:374] T 2bdcdcf5f8d14c608d7d9d445f993b26 P 3559026efbc642419cabdda2898714a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:00:08.826615 15537 raft_consensus.cc:729] T 2bdcdcf5f8d14c608d7d9d445f993b26 P 3559026efbc642419cabdda2898714a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3559026efbc642419cabdda2898714a6, State: Initialized, Role: FOLLOWER
I20260928 15:00:08.827358 15537 consensus_queue.cc:260] T 2bdcdcf5f8d14c608d7d9d445f993b26 P 3559026efbc642419cabdda2898714a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b9cbb8fe0945beb891b825281d5bfb" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 46849 } } peers { permanent_uuid: "3559026efbc642419cabdda2898714a6" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 32855 } } peers { permanent_uuid: "14b53f4d618d4bfb96ce8b6fc0746dc0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 32837 } }
I20260928 15:00:08.829005 15383 heartbeater.cc:503] Master 127.31.40.62:34035 was elected leader, sending a full tablet report...
I20260928 15:00:08.829797 15537 ts_tablet_manager.cc:1428] T 2bdcdcf5f8d14c608d7d9d445f993b26 P 3559026efbc642419cabdda2898714a6: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260928 15:00:08.829816 15539 raft_consensus.cc:348] T 2bdcdcf5f8d14c608d7d9d445f993b26 P 14b53f4d618d4bfb96ce8b6fc0746dc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b9cbb8fe0945beb891b825281d5bfb" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 46849 } } peers { permanent_uuid: "3559026efbc642419cabdda2898714a6" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 32855 } } peers { permanent_uuid: "14b53f4d618d4bfb96ce8b6fc0746dc0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 32837 } }
I20260928 15:00:08.830267 15539 raft_consensus.cc:374] T 2bdcdcf5f8d14c608d7d9d445f993b26 P 14b53f4d618d4bfb96ce8b6fc0746dc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:00:08.830372 15539 raft_consensus.cc:729] T 2bdcdcf5f8d14c608d7d9d445f993b26 P 14b53f4d618d4bfb96ce8b6fc0746dc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14b53f4d618d4bfb96ce8b6fc0746dc0, State: Initialized, Role: FOLLOWER
I20260928 15:00:08.831105 15539 consensus_queue.cc:260] T 2bdcdcf5f8d14c608d7d9d445f993b26 P 14b53f4d618d4bfb96ce8b6fc0746dc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b9cbb8fe0945beb891b825281d5bfb" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 46849 } } peers { permanent_uuid: "3559026efbc642419cabdda2898714a6" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 32855 } } peers { permanent_uuid: "14b53f4d618d4bfb96ce8b6fc0746dc0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 32837 } }
I20260928 15:00:08.832736 15539 ts_tablet_manager.cc:1428] T 2bdcdcf5f8d14c608d7d9d445f993b26 P 14b53f4d618d4bfb96ce8b6fc0746dc0: Time spent starting tablet: real 0.014s user 0.008s sys 0.006s
W20260928 15:00:08.872107 15250 tablet.cc:2369] T 2bdcdcf5f8d14c608d7d9d445f993b26 P 14b53f4d618d4bfb96ce8b6fc0746dc0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:00:08.911226 15384 tablet.cc:2369] T 2bdcdcf5f8d14c608d7d9d445f993b26 P 3559026efbc642419cabdda2898714a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:00:08.940707 15518 tablet.cc:2369] T 2bdcdcf5f8d14c608d7d9d445f993b26 P e3b9cbb8fe0945beb891b825281d5bfb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:00:09.040998 15544 raft_consensus.cc:482] T 2bdcdcf5f8d14c608d7d9d445f993b26 P 14b53f4d618d4bfb96ce8b6fc0746dc0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:00:09.041342 15544 raft_consensus.cc:504] T 2bdcdcf5f8d14c608d7d9d445f993b26 P 14b53f4d618d4bfb96ce8b6fc0746dc0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b9cbb8fe0945beb891b825281d5bfb" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 46849 } } peers { permanent_uuid: "3559026efbc642419cabdda2898714a6" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 32855 } } peers { permanent_uuid: "14b53f4d618d4bfb96ce8b6fc0746dc0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 32837 } }
I20260928 15:00:09.043192 15544 leader_election.cc:288] T 2bdcdcf5f8d14c608d7d9d445f993b26 P 14b53f4d618d4bfb96ce8b6fc0746dc0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3b9cbb8fe0945beb891b825281d5bfb (127.31.40.3:46849), 3559026efbc642419cabdda2898714a6 (127.31.40.2:32855)
I20260928 15:00:09.052783 15532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:09.043812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:42035 (local address 127.31.40.3:46849)
0928 15:00:09.044080 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:09.044091 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:09.044745 (+ 654us) server_negotiation.cc:407] Connection header received
0928 15:00:09.044939 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:09.044945 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:09.045026 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:09.045128 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:00:09.046069 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:09.047188 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:09.050523 (+ 3335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:09.051813 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:09.051976 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:09.052082 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:09.052385 (+ 303us) server_negotiation.cc:299] Negotiation successful
0928 15:00:09.052515 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 15:00:09.052911 15545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:09.043615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.3:46849 (local address 127.31.40.1:42035)
0928 15:00:09.044645 (+ 1030us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:09.044662 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:09.044783 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:09.045174 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:09.045180 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:09.045214 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:00:09.045881 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:09.045897 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:09.047359 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:09.047365 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:09.050335 (+ 2970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:09.050352 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:09.052061 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:09.052066 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:09.052191 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:09.052217 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 15:00:09.052315 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 15:00:09.052415 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":904,"mutex_wait_us":153,"thread_start_us":499,"threads_started":1}
I20260928 15:00:09.053289 15546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:09.043575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.2:32855 (local address 127.31.40.1:50859)
0928 15:00:09.044500 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:09.044535 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:09.044719 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:09.045174 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:09.045180 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:09.045237 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:00:09.045887 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:09.045897 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:09.047359 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:09.047365 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:09.050341 (+ 2976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:09.050352 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:09.052248 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:09.052253 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:09.052455 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:09.052489 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 15:00:09.052590 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 15:00:09.052714 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"mutex_wait_us":130,"spinlock_wait_cycles":5376,"thread_start_us":736,"threads_started":1}
I20260928 15:00:09.053418 15533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:09.043838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:50859 (local address 127.31.40.2:32855)
0928 15:00:09.044106 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:09.044114 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:09.044649 (+ 535us) server_negotiation.cc:407] Connection header received
0928 15:00:09.044940 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:09.044945 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:09.045026 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:09.045128 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:00:09.046074 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:09.047188 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:09.050523 (+ 3335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:09.051813 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:09.051979 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:09.052079 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:09.052695 (+ 616us) server_negotiation.cc:299] Negotiation successful
0928 15:00:09.052868 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260928 15:00:09.054024 15472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bdcdcf5f8d14c608d7d9d445f993b26" candidate_uuid: "14b53f4d618d4bfb96ce8b6fc0746dc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3b9cbb8fe0945beb891b825281d5bfb" is_pre_election: true
I20260928 15:00:09.054350 15338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bdcdcf5f8d14c608d7d9d445f993b26" candidate_uuid: "14b53f4d618d4bfb96ce8b6fc0746dc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3559026efbc642419cabdda2898714a6" is_pre_election: true
I20260928 15:00:09.054612 15472 raft_consensus.cc:2454] T 2bdcdcf5f8d14c608d7d9d445f993b26 P e3b9cbb8fe0945beb891b825281d5bfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14b53f4d618d4bfb96ce8b6fc0746dc0 in term 0.
I20260928 15:00:09.054912 15338 raft_consensus.cc:2454] T 2bdcdcf5f8d14c608d7d9d445f993b26 P 3559026efbc642419cabdda2898714a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14b53f4d618d4bfb96ce8b6fc0746dc0 in term 0.
I20260928 15:00:09.055394 15137 leader_election.cc:302] T 2bdcdcf5f8d14c608d7d9d445f993b26 P 14b53f4d618d4bfb96ce8b6fc0746dc0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14b53f4d618d4bfb96ce8b6fc0746dc0, e3b9cbb8fe0945beb891b825281d5bfb; no voters:
I20260928 15:00:09.055830 15544 raft_consensus.cc:2781] T 2bdcdcf5f8d14c608d7d9d445f993b26 P 14b53f4d618d4bfb96ce8b6fc0746dc0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:00:09.056007 15544 raft_consensus.cc:482] T 2bdcdcf5f8d14c608d7d9d445f993b26 P 14b53f4d618d4bfb96ce8b6fc0746dc0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:00:09.056135 15544 raft_consensus.cc:3037] T 2bdcdcf5f8d14c608d7d9d445f993b26 P 14b53f4d618d4bfb96ce8b6fc0746dc0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:00:09.058337 15544 raft_consensus.cc:504] T 2bdcdcf5f8d14c608d7d9d445f993b26 P 14b53f4d618d4bfb96ce8b6fc0746dc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b9cbb8fe0945beb891b825281d5bfb" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 46849 } } peers { permanent_uuid: "3559026efbc642419cabdda2898714a6" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 32855 } } peers { permanent_uuid: "14b53f4d618d4bfb96ce8b6fc0746dc0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 32837 } }
I20260928 15:00:09.058948 15544 leader_election.cc:288] T 2bdcdcf5f8d14c608d7d9d445f993b26 P 14b53f4d618d4bfb96ce8b6fc0746dc0 [CANDIDATE]: Term 1 election: Requested vote from peers e3b9cbb8fe0945beb891b825281d5bfb (127.31.40.3:46849), 3559026efbc642419cabdda2898714a6 (127.31.40.2:32855)
I20260928 15:00:09.059446 15338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bdcdcf5f8d14c608d7d9d445f993b26" candidate_uuid: "14b53f4d618d4bfb96ce8b6fc0746dc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3559026efbc642419cabdda2898714a6"
I20260928 15:00:09.059470 15472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bdcdcf5f8d14c608d7d9d445f993b26" candidate_uuid: "14b53f4d618d4bfb96ce8b6fc0746dc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3b9cbb8fe0945beb891b825281d5bfb"
I20260928 15:00:09.059661 15338 raft_consensus.cc:3037] T 2bdcdcf5f8d14c608d7d9d445f993b26 P 3559026efbc642419cabdda2898714a6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:00:09.059664 15472 raft_consensus.cc:3037] T 2bdcdcf5f8d14c608d7d9d445f993b26 P e3b9cbb8fe0945beb891b825281d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:00:09.061949 15472 raft_consensus.cc:2454] T 2bdcdcf5f8d14c608d7d9d445f993b26 P e3b9cbb8fe0945beb891b825281d5bfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14b53f4d618d4bfb96ce8b6fc0746dc0 in term 1.
I20260928 15:00:09.061952 15338 raft_consensus.cc:2454] T 2bdcdcf5f8d14c608d7d9d445f993b26 P 3559026efbc642419cabdda2898714a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14b53f4d618d4bfb96ce8b6fc0746dc0 in term 1.
I20260928 15:00:09.062395 15137 leader_election.cc:302] T 2bdcdcf5f8d14c608d7d9d445f993b26 P 14b53f4d618d4bfb96ce8b6fc0746dc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 14b53f4d618d4bfb96ce8b6fc0746dc0, 3559026efbc642419cabdda2898714a6, e3b9cbb8fe0945beb891b825281d5bfb; no voters:
I20260928 15:00:09.062731 15544 raft_consensus.cc:2781] T 2bdcdcf5f8d14c608d7d9d445f993b26 P 14b53f4d618d4bfb96ce8b6fc0746dc0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:00:09.063246 15544 raft_consensus.cc:686] T 2bdcdcf5f8d14c608d7d9d445f993b26 P 14b53f4d618d4bfb96ce8b6fc0746dc0 [term 1 LEADER]: Becoming Leader. State: Replica: 14b53f4d618d4bfb96ce8b6fc0746dc0, State: Running, Role: LEADER
I20260928 15:00:09.063750 15544 consensus_queue.cc:237] T 2bdcdcf5f8d14c608d7d9d445f993b26 P 14b53f4d618d4bfb96ce8b6fc0746dc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b9cbb8fe0945beb891b825281d5bfb" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 46849 } } peers { permanent_uuid: "3559026efbc642419cabdda2898714a6" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 32855 } } peers { permanent_uuid: "14b53f4d618d4bfb96ce8b6fc0746dc0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 32837 } }
I20260928 15:00:09.068897 15060 catalog_manager.cc:5495] T 2bdcdcf5f8d14c608d7d9d445f993b26 P 14b53f4d618d4bfb96ce8b6fc0746dc0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 14b53f4d618d4bfb96ce8b6fc0746dc0 (127.31.40.1). New cstate: current_term: 1 leader_uuid: "14b53f4d618d4bfb96ce8b6fc0746dc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b9cbb8fe0945beb891b825281d5bfb" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 46849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3559026efbc642419cabdda2898714a6" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 32855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14b53f4d618d4bfb96ce8b6fc0746dc0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 32837 } health_report { overall_health: HEALTHY } } }
I20260928 15:00:09.156852 15535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:09.150702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44976 (local address 127.31.40.1:32837)
0928 15:00:09.150947 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:09.150953 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:09.150975 (+ 22us) server_negotiation.cc:407] Connection header received
0928 15:00:09.151255 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:09.151259 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:09.151368 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:09.151535 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:00:09.152006 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:09.152864 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:09.154441 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:09.154873 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:09.155013 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:09.155591 (+ 578us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:00:09.156170 (+ 579us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:00:09.156341 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:09.156422 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 15:00:09.156516 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 15:00:09.165954 15338 raft_consensus.cc:1261] T 2bdcdcf5f8d14c608d7d9d445f993b26 P 3559026efbc642419cabdda2898714a6 [term 1 FOLLOWER]: Refusing update from remote peer 14b53f4d618d4bfb96ce8b6fc0746dc0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:00:09.165954 15472 raft_consensus.cc:1261] T 2bdcdcf5f8d14c608d7d9d445f993b26 P e3b9cbb8fe0945beb891b825281d5bfb [term 1 FOLLOWER]: Refusing update from remote peer 14b53f4d618d4bfb96ce8b6fc0746dc0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:00:09.167097 15544 consensus_queue.cc:1035] T 2bdcdcf5f8d14c608d7d9d445f993b26 P 14b53f4d618d4bfb96ce8b6fc0746dc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3559026efbc642419cabdda2898714a6" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 32855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:00:09.167412 15547 consensus_queue.cc:1035] T 2bdcdcf5f8d14c608d7d9d445f993b26 P 14b53f4d618d4bfb96ce8b6fc0746dc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b9cbb8fe0945beb891b825281d5bfb" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 46849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:00:09.200307 15060 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:00:09.200584 15060 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:00:09.204270 15060 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69: Sending DeleteTablet for 3 replicas of tablet 2bdcdcf5f8d14c608d7d9d445f993b26
I20260928 15:00:09.205488 15452 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bdcdcf5f8d14c608d7d9d445f993b26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:00:09 UTC) from {username='kudu', principal='kudu/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:43006
I20260928 15:00:09.205519 15318 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bdcdcf5f8d14c608d7d9d445f993b26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:00:09 UTC) from {username='kudu', principal='kudu/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:57750
I20260928 15:00:09.205874 15184 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bdcdcf5f8d14c608d7d9d445f993b26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:00:09 UTC) from {username='kudu', principal='kudu/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:44962
I20260928 15:00:09.207229 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 15121
I20260928 15:00:09.214915 15562 tablet_replica.cc:330] T 2bdcdcf5f8d14c608d7d9d445f993b26 P 14b53f4d618d4bfb96ce8b6fc0746dc0: stopping tablet replica
I20260928 15:00:09.215497 15562 raft_consensus.cc:2229] T 2bdcdcf5f8d14c608d7d9d445f993b26 P 14b53f4d618d4bfb96ce8b6fc0746dc0 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:00:09.216230 15561 tablet_replica.cc:330] T 2bdcdcf5f8d14c608d7d9d445f993b26 P 3559026efbc642419cabdda2898714a6: stopping tablet replica
I20260928 15:00:09.216329 15562 raft_consensus.cc:2258] T 2bdcdcf5f8d14c608d7d9d445f993b26 P 14b53f4d618d4bfb96ce8b6fc0746dc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:00:09.216629 15560 tablet_replica.cc:330] T 2bdcdcf5f8d14c608d7d9d445f993b26 P e3b9cbb8fe0945beb891b825281d5bfb: stopping tablet replica
I20260928 15:00:09.217778 15561 raft_consensus.cc:2229] T 2bdcdcf5f8d14c608d7d9d445f993b26 P 3559026efbc642419cabdda2898714a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:00:09.218115 15560 raft_consensus.cc:2229] T 2bdcdcf5f8d14c608d7d9d445f993b26 P e3b9cbb8fe0945beb891b825281d5bfb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:00:09.218652 15562 ts_tablet_manager.cc:1905] T 2bdcdcf5f8d14c608d7d9d445f993b26 P 14b53f4d618d4bfb96ce8b6fc0746dc0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:00:09.219189 15561 raft_consensus.cc:2258] T 2bdcdcf5f8d14c608d7d9d445f993b26 P 3559026efbc642419cabdda2898714a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:00:09.218688 15560 raft_consensus.cc:2258] T 2bdcdcf5f8d14c608d7d9d445f993b26 P e3b9cbb8fe0945beb891b825281d5bfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:00:09.221860 15560 ts_tablet_manager.cc:1905] T 2bdcdcf5f8d14c608d7d9d445f993b26 P e3b9cbb8fe0945beb891b825281d5bfb: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 15:00:09 dist-test-slave-j968 krb5kdc[15015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.1@KRBTEST.COM
I20260928 15:00:09.228552 15561 ts_tablet_manager.cc:1905] T 2bdcdcf5f8d14c608d7d9d445f993b26 P 3559026efbc642419cabdda2898714a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:00:09.229274 15560 ts_tablet_manager.cc:1918] T 2bdcdcf5f8d14c608d7d9d445f993b26 P e3b9cbb8fe0945beb891b825281d5bfb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:00:09.229542 15560 log.cc:1196] T 2bdcdcf5f8d14c608d7d9d445f993b26 P e3b9cbb8fe0945beb891b825281d5bfb: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-2/wal/wals/2bdcdcf5f8d14c608d7d9d445f993b26
I20260928 15:00:09.230114 15560 ts_tablet_manager.cc:1939] T 2bdcdcf5f8d14c608d7d9d445f993b26 P e3b9cbb8fe0945beb891b825281d5bfb: Deleting consensus metadata
I20260928 15:00:09.232216 15044 catalog_manager.cc:4841] TS e3b9cbb8fe0945beb891b825281d5bfb (127.31.40.3:46849): tablet 2bdcdcf5f8d14c608d7d9d445f993b26 (table test-table [id=5acfdf07a48c4dbd9257bb468b95b7d8]) successfully deleted
I20260928 15:00:09.234727 15561 ts_tablet_manager.cc:1918] T 2bdcdcf5f8d14c608d7d9d445f993b26 P 3559026efbc642419cabdda2898714a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:00:09.234938 15561 log.cc:1196] T 2bdcdcf5f8d14c608d7d9d445f993b26 P 3559026efbc642419cabdda2898714a6: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-1/wal/wals/2bdcdcf5f8d14c608d7d9d445f993b26
I20260928 15:00:09.235127 15562 ts_tablet_manager.cc:1918] T 2bdcdcf5f8d14c608d7d9d445f993b26 P 14b53f4d618d4bfb96ce8b6fc0746dc0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:00:09.235371 15562 log.cc:1196] T 2bdcdcf5f8d14c608d7d9d445f993b26 P 14b53f4d618d4bfb96ce8b6fc0746dc0: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-0/wal/wals/2bdcdcf5f8d14c608d7d9d445f993b26
I20260928 15:00:09.235517 15561 ts_tablet_manager.cc:1939] T 2bdcdcf5f8d14c608d7d9d445f993b26 P 3559026efbc642419cabdda2898714a6: Deleting consensus metadata
I20260928 15:00:09.236066 15562 ts_tablet_manager.cc:1939] T 2bdcdcf5f8d14c608d7d9d445f993b26 P 14b53f4d618d4bfb96ce8b6fc0746dc0: Deleting consensus metadata
I20260928 15:00:09.237545 15047 catalog_manager.cc:4841] TS 3559026efbc642419cabdda2898714a6 (127.31.40.2:32855): tablet 2bdcdcf5f8d14c608d7d9d445f993b26 (table test-table [id=5acfdf07a48c4dbd9257bb468b95b7d8]) successfully deleted
I20260928 15:00:09.237814 15045 catalog_manager.cc:4841] TS 14b53f4d618d4bfb96ce8b6fc0746dc0 (127.31.40.1:32837): tablet 2bdcdcf5f8d14c608d7d9d445f993b26 (table test-table [id=5acfdf07a48c4dbd9257bb468b95b7d8]) successfully deleted
I20260928 15:00:09.238626 15535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:09.208126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44990 (local address 127.31.40.1:32837)
0928 15:00:09.208319 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:09.208325 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:09.215567 (+ 7242us) server_negotiation.cc:407] Connection header received
0928 15:00:09.216321 (+ 754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:09.216328 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:09.216417 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:09.216503 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:00:09.218748 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:09.219984 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:09.222116 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:09.222908 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:09.223026 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:09.230042 (+ 7016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:00:09.230062 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:00:09.230082 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:00:09.230143 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:00:09.234613 (+ 4470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:09.235694 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:09.235704 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:09.235708 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:09.235943 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:09.236467 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:09.236472 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:09.236475 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:09.236995 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:00:09.237297 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:09.237981 (+ 684us) server_negotiation.cc:299] Negotiation successful
0928 15:00:09.238237 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 15:00:09.240095 15244 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:00:09.341792 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 15121
I20260928 15:00:09.359246 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 15255
Sep 28 15:00:09 dist-test-slave-j968 krb5kdc[15015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.2@KRBTEST.COM
I20260928 15:00:09.375646 15533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:09.360277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57764 (local address 127.31.40.2:32855)
0928 15:00:09.360476 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:09.360484 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:09.360509 (+ 25us) server_negotiation.cc:407] Connection header received
0928 15:00:09.360584 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:09.360588 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:09.360657 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:09.360766 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:00:09.362060 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:09.362793 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:09.364267 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:09.364682 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:09.364818 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:09.369784 (+ 4966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:00:09.369804 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:00:09.369823 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:00:09.369865 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:00:09.372931 (+ 3066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:09.373666 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:09.373674 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:09.373677 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:09.373845 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:09.374181 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:09.374184 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:09.374186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:09.374591 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:00:09.374857 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:09.375164 (+ 307us) server_negotiation.cc:299] Negotiation successful
0928 15:00:09.375371 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 15:00:09.376417 15378 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:00:09.468967 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 15255
I20260928 15:00:09.487582 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 15389
Sep 28 15:00:09 dist-test-slave-j968 krb5kdc[15015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.3@KRBTEST.COM
I20260928 15:00:09.504056 15532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:09.488432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43016 (local address 127.31.40.3:46849)
0928 15:00:09.488635 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:09.488640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:09.488663 (+ 23us) server_negotiation.cc:407] Connection header received
0928 15:00:09.488943 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:09.488947 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:09.489018 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:09.489147 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:00:09.490565 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:09.491287 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:09.492830 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:09.493220 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:09.493357 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:09.497982 (+ 4625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:00:09.497998 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:00:09.498014 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:00:09.498050 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:00:09.501095 (+ 3045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:09.501937 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:09.501945 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:09.501947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:09.502096 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:09.502467 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:09.502472 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:09.502475 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:09.502856 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:00:09.503158 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:09.503451 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 15:00:09.503685 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 15:00:09.504743 15512 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:00:09.596283 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 15389
I20260928 15:00:09.615586 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 15028
I20260928 15:00:09.616837 15090 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:00:09.733093 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 15028
I20260928 15:00:29.756582 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:34035
--webserver_interface=127.31.40.62
--webserver_port=36777
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:34399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.40.62:34035
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:00:30.098640 15568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:00:30.098937 15568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:00:30.099061 15568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:00:30.106503 15568 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 15:00:30.106618 15568 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 15:00:30.106658 15568 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 15:00:30.106752 15568 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:00:30.106848 15568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:00:30.106941 15568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:00:30.107033 15568 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:00:30.107112 15568 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:00:30.117252 15568 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:34399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.31.40.62:34035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:34035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.40.62
--webserver_port=36777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:00:30.118654 15568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:00:30.120137 15568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:00:30.146529 15577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:30.146485 15575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:30.146467 15574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:00:30.146832 15568 server_base.cc:1034] running on GCE node
I20260928 15:00:30.148212 15568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:00:30.149848 15568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:00:30.151114 15568 hybrid_clock.cc:648] HybridClock initialized: now 1790607630151049 us; error 37 us; skew 500 ppm
I20260928 15:00:30.151526 15568 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:00:30 dist-test-slave-j968 krb5kdc[15015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607630, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:00:30.155311 15568 init.cc:377] Logged in from keytab as kudu/127.31.40.62@KRBTEST.COM (short username kudu)
I20260928 15:00:30.157090 15568 webserver.cc:460] Webserver started at http://127.31.40.62:36777/ using document root <none> and password file <none>
I20260928 15:00:30.157891 15568 fs_manager.cc:359] Metadata directory not provided
I20260928 15:00:30.158048 15568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:00:30.164280 15568 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 15:00:30.167268 15584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:30.168692 15568 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:00:30.168888 15568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/master-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/master-0/wal
uuid: "99120398a8b0443aa9313e4874f4aa69"
format_stamp: "Formatted at 2026-09-28 15:00:06 on dist-test-slave-j968"
server_key: "c032cd757ec4b7ee1e8e39128856c2ea"
server_key_iv: "84998d5837221dcbb3e30753c62ad036"
server_key_version: "encryptionkey@0"
I20260928 15:00:30.169968 15568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:00:30.209517 15568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:00:30.214687 15568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:00:30.215031 15568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:00:30.234983 15568 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.62:34035
I20260928 15:00:30.235073 15635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.62:34035 every 8 connection(s)
I20260928 15:00:30.236620 15568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
I20260928 15:00:30.242086 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 15568
I20260928 15:00:30.242944 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.1:32837
--local_ip_for_outbound_sockets=127.31.40.1
--tserver_master_addrs=127.31.40.62:34035
--webserver_port=38983
--webserver_interface=127.31.40.1
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab
--principal=kudu/127.31.40.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:34399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:00:30.244130 15636 sys_catalog.cc:263] Verifying existing consensus state
I20260928 15:00:30.247465 15636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69: Bootstrap starting.
I20260928 15:00:30.275419 15636 log.cc:824] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69: Log is configured to *not* fsync() on all Append() calls
I20260928 15:00:30.291179 15636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 15:00:30.291843 15636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69: Bootstrap complete.
I20260928 15:00:30.305943 15636 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99120398a8b0443aa9313e4874f4aa69" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34035 } }
I20260928 15:00:30.307173 15636 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99120398a8b0443aa9313e4874f4aa69, State: Initialized, Role: FOLLOWER
I20260928 15:00:30.307997 15636 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99120398a8b0443aa9313e4874f4aa69" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34035 } }
I20260928 15:00:30.308301 15636 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:00:30.308434 15636 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:00:30.308657 15636 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69 [term 1 FOLLOWER]: Advancing to term 2
I20260928 15:00:30.312631 15636 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99120398a8b0443aa9313e4874f4aa69" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34035 } }
I20260928 15:00:30.313194 15636 leader_election.cc:302] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99120398a8b0443aa9313e4874f4aa69; no voters:
I20260928 15:00:30.313903 15636 leader_election.cc:288] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 15:00:30.314052 15640 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 15:00:30.315291 15640 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69 [term 2 LEADER]: Becoming Leader. State: Replica: 99120398a8b0443aa9313e4874f4aa69, State: Running, Role: LEADER
I20260928 15:00:30.315995 15640 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99120398a8b0443aa9313e4874f4aa69" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34035 } }
I20260928 15:00:30.316463 15636 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:00:30.318873 15642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "99120398a8b0443aa9313e4874f4aa69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99120398a8b0443aa9313e4874f4aa69" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34035 } } }
I20260928 15:00:30.319257 15642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69 [sys.catalog]: This master's current role is: LEADER
I20260928 15:00:30.319149 15641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99120398a8b0443aa9313e4874f4aa69. Latest consensus state: current_term: 2 leader_uuid: "99120398a8b0443aa9313e4874f4aa69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99120398a8b0443aa9313e4874f4aa69" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34035 } } }
I20260928 15:00:30.319659 15641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69 [sys.catalog]: This master's current role is: LEADER
I20260928 15:00:30.329741 15647 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:00:30.338418 15647 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:00:30.340413 15647 catalog_manager.cc:1259] Loaded cluster ID: bfc018e2a8b7437587bf5b8abf7d1632
I20260928 15:00:30.340612 15647 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:00:30.346159 15647 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:00:30.348441 15647 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69: Loaded TSK: 0
I20260928 15:00:30.380055 15647 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69: Generated new TSK 1
I20260928 15:00:30.380519 15647 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 15:00:30.647069 15638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:00:30.647377 15638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:00:30.647531 15638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:00:30.654996 15638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:00:30.655113 15638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:00:30.655290 15638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.1
I20260928 15:00:30.665407 15638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:34399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab
--principal=kudu/127.31.40.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.1:32837
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.40.1
--webserver_port=38983
--tserver_master_addrs=127.31.40.62:34035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.1
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:00:30.666939 15638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:00:30.668382 15638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:00:30.684092 15663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:30.684077 15664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:30.684748 15666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:00:30.685904 15638 server_base.cc:1034] running on GCE node
I20260928 15:00:30.687000 15638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:00:30.688418 15638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:00:30.689707 15638 hybrid_clock.cc:648] HybridClock initialized: now 1790607630689639 us; error 54 us; skew 500 ppm
I20260928 15:00:30.690132 15638 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:00:30 dist-test-slave-j968 krb5kdc[15015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607630, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:00:30.700990 15638 init.cc:377] Logged in from keytab as kudu/127.31.40.1@KRBTEST.COM (short username kudu)
I20260928 15:00:30.702960 15638 webserver.cc:460] Webserver started at http://127.31.40.1:38983/ using document root <none> and password file <none>
I20260928 15:00:30.703850 15638 fs_manager.cc:359] Metadata directory not provided
I20260928 15:00:30.704033 15638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:00:30.711154 15638 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260928 15:00:30.714674 15673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:30.716502 15638 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 15:00:30.716693 15638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-0/wal
uuid: "14b53f4d618d4bfb96ce8b6fc0746dc0"
format_stamp: "Formatted at 2026-09-28 15:00:07 on dist-test-slave-j968"
server_key: "ee4d4c5d194c7304fecefb3eef4fd5e2"
server_key_iv: "f6b65b0ead875215fb8c94b737eab460"
server_key_version: "encryptionkey@0"
I20260928 15:00:30.718181 15638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:00:30.751062 15638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:00:30.756409 15638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:00:30.756721 15638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:00:30.757968 15638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:00:30.759635 15638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:00:30.759730 15638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:30.759810 15638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:00:30.759931 15638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:30.795094 15638 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.1:32837
I20260928 15:00:30.795147 15785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.1:32837 every 8 connection(s)
I20260928 15:00:30.796696 15638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
I20260928 15:00:30.803686 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 15638
I20260928 15:00:30.804483 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.2:32855
--local_ip_for_outbound_sockets=127.31.40.2
--tserver_master_addrs=127.31.40.62:34035
--webserver_port=36365
--webserver_interface=127.31.40.2
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab
--principal=kudu/127.31.40.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:34399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 15:00:30 dist-test-slave-j968 krb5kdc[15015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607630, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.1@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 15:00:30.821844 15788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:30.799733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:49517 (local address 127.31.40.62:34035)
0928 15:00:30.800340 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:30.800356 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:30.801049 (+ 693us) server_negotiation.cc:407] Connection header received
0928 15:00:30.803262 (+ 2213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:30.803275 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:30.803634 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:30.804070 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:00:30.805797 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:30.807087 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:30.808743 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:30.809438 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:30.809653 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:30.813679 (+ 4026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:00:30.813687 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:00:30.813705 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:00:30.813752 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:00:30.816991 (+ 3239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:30.817796 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:30.817805 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:30.817814 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:30.818005 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:30.818482 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:30.818489 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:30.818496 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:30.818985 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:00:30.819203 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:30.820052 (+ 849us) server_negotiation.cc:299] Negotiation successful
0928 15:00:30.820484 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":296,"threads_started":1}
I20260928 15:00:30.821844 15789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:30.800016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34035 (local address 127.31.40.1:49517)
0928 15:00:30.800861 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:30.800907 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:30.802698 (+ 1791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:30.804205 (+ 1507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:30.804220 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:30.804970 (+ 750us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:00:30.805543 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:30.805561 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:30.807277 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:30.807287 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:30.808590 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:30.808603 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:30.809693 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:30.809701 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:30.809992 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:30.810777 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:00:30.810805 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:00:30.813369 (+ 2564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:00:30.817154 (+ 3785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:30.817165 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:30.817182 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:30.817632 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:30.818143 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:30.818148 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:30.818152 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:30.818350 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:30.819129 (+ 779us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:00:30.819137 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:00:30.819502 (+ 365us) client_negotiation.cc:769] Sending connection context
0928 15:00:30.819780 (+ 278us) client_negotiation.cc:240] Negotiation successful
0928 15:00:30.820408 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":360,"threads_started":1}
I20260928 15:00:30.825743 15786 heartbeater.cc:347] Connected to a master server at 127.31.40.62:34035
I20260928 15:00:30.826112 15786 heartbeater.cc:464] Registering TS with master...
I20260928 15:00:30.827030 15786 heartbeater.cc:511] Master 127.31.40.62:34035 requested a full tablet report, sending...
I20260928 15:00:30.829533 15601 ts_manager.cc:194] Registered new tserver with Master: 14b53f4d618d4bfb96ce8b6fc0746dc0 (127.31.40.1:32837)
I20260928 15:00:30.831226 15601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.1@KRBTEST.COM'} at 127.31.40.1:49517
W20260928 15:00:31.150997 15790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:00:31.151298 15790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:00:31.151467 15790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:00:31.159039 15790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:00:31.159153 15790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:00:31.159327 15790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.2
I20260928 15:00:31.170136 15790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:34399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab
--principal=kudu/127.31.40.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.2:32855
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.40.2
--webserver_port=36365
--tserver_master_addrs=127.31.40.62:34035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.2
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:00:31.171525 15790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:00:31.172933 15790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:00:31.186189 15796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:31.186188 15798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:31.186460 15795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:00:31.187043 15790 server_base.cc:1034] running on GCE node
I20260928 15:00:31.187703 15790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:00:31.188792 15790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:00:31.190056 15790 hybrid_clock.cc:648] HybridClock initialized: now 1790607631189991 us; error 50 us; skew 500 ppm
I20260928 15:00:31.190409 15790 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:00:31 dist-test-slave-j968 krb5kdc[15015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607631, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:00:31.193737 15790 init.cc:377] Logged in from keytab as kudu/127.31.40.2@KRBTEST.COM (short username kudu)
I20260928 15:00:31.195225 15790 webserver.cc:460] Webserver started at http://127.31.40.2:36365/ using document root <none> and password file <none>
I20260928 15:00:31.195986 15790 fs_manager.cc:359] Metadata directory not provided
I20260928 15:00:31.196142 15790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:00:31.201428 15790 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:00:31.204286 15805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:31.205623 15790 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 15:00:31.205803 15790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-1/wal
uuid: "3559026efbc642419cabdda2898714a6"
format_stamp: "Formatted at 2026-09-28 15:00:08 on dist-test-slave-j968"
server_key: "bcb92fd637f53804a8d6c6b0460b1226"
server_key_iv: "3c0fd12e9261b1e47506f8167fe68de4"
server_key_version: "encryptionkey@0"
I20260928 15:00:31.206807 15790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:00:31.230660 15790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:00:31.235298 15790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:00:31.235620 15790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:00:31.236716 15790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:00:31.238121 15790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:00:31.238253 15790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:31.238394 15790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:00:31.238487 15790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:31.272619 15790 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.2:32855
I20260928 15:00:31.272708 15917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.2:32855 every 8 connection(s)
I20260928 15:00:31.274116 15790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
I20260928 15:00:31.280792 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 15790
I20260928 15:00:31.281585 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.3:46849
--local_ip_for_outbound_sockets=127.31.40.3
--tserver_master_addrs=127.31.40.62:34035
--webserver_port=33821
--webserver_interface=127.31.40.3
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab
--principal=kudu/127.31.40.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:34399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 15:00:31 dist-test-slave-j968 krb5kdc[15015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607631, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.2@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 15:00:31.296126 15788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:31.276825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:49155 (local address 127.31.40.62:34035)
0928 15:00:31.277071 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:31.277077 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:31.278151 (+ 1074us) server_negotiation.cc:407] Connection header received
0928 15:00:31.279969 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:31.279973 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:31.280054 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:31.280140 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:00:31.282225 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:31.282952 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:31.285397 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:31.285828 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:31.285926 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:31.290391 (+ 4465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:00:31.290396 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:00:31.290400 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:00:31.290436 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:00:31.293053 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:31.293763 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:31.293769 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:31.293773 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:31.293902 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:31.294324 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:31.294329 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:31.294332 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:31.294622 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:00:31.294775 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:31.295498 (+ 723us) server_negotiation.cc:299] Negotiation successful
0928 15:00:31.295734 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260928 15:00:31.297133 15920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:31.277298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34035 (local address 127.31.40.2:49155)
0928 15:00:31.277990 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:31.278028 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:31.279670 (+ 1642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:31.280400 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:31.280439 (+ 39us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:31.281207 (+ 768us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:00:31.282052 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:31.282080 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:31.283114 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:31.283122 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:31.285253 (+ 2131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:31.285271 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:31.286067 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:31.286077 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:31.286418 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:31.287209 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:00:31.287249 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:00:31.290141 (+ 2892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:00:31.293190 (+ 3049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:31.293199 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:31.293216 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:31.293641 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:31.294018 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:31.294024 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:31.294028 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:31.294213 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:31.294749 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:00:31.294757 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:00:31.295143 (+ 386us) client_negotiation.cc:769] Sending connection context
0928 15:00:31.295424 (+ 281us) client_negotiation.cc:240] Negotiation successful
0928 15:00:31.295998 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":263,"threads_started":1}
I20260928 15:00:31.298740 15918 heartbeater.cc:347] Connected to a master server at 127.31.40.62:34035
I20260928 15:00:31.299060 15918 heartbeater.cc:464] Registering TS with master...
I20260928 15:00:31.299783 15918 heartbeater.cc:511] Master 127.31.40.62:34035 requested a full tablet report, sending...
I20260928 15:00:31.301398 15601 ts_manager.cc:194] Registered new tserver with Master: 3559026efbc642419cabdda2898714a6 (127.31.40.2:32855)
I20260928 15:00:31.302757 15601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.2@KRBTEST.COM'} at 127.31.40.2:49155
W20260928 15:00:31.620303 15921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:00:31.620574 15921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:00:31.620671 15921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:00:31.628235 15921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:00:31.628348 15921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:00:31.628468 15921 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.3
I20260928 15:00:31.638226 15921 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:34399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab
--principal=kudu/127.31.40.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.3:46849
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.40.3
--webserver_port=33821
--tserver_master_addrs=127.31.40.62:34035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.3
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:00:31.639505 15921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:00:31.640794 15921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:00:31.654191 15926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:31.654310 15929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:31.654337 15927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:00:31.655637 15921 server_base.cc:1034] running on GCE node
I20260928 15:00:31.656334 15921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:00:31.657295 15921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:00:31.658529 15921 hybrid_clock.cc:648] HybridClock initialized: now 1790607631658496 us; error 58 us; skew 500 ppm
I20260928 15:00:31.658851 15921 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:00:31 dist-test-slave-j968 krb5kdc[15015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607631, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:00:31.662000 15921 init.cc:377] Logged in from keytab as kudu/127.31.40.3@KRBTEST.COM (short username kudu)
I20260928 15:00:31.663453 15921 webserver.cc:460] Webserver started at http://127.31.40.3:33821/ using document root <none> and password file <none>
I20260928 15:00:31.664209 15921 fs_manager.cc:359] Metadata directory not provided
I20260928 15:00:31.664363 15921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:00:31.669562 15921 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 15:00:31.672363 15936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:31.673877 15921 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:00:31.674075 15921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-2/wal
uuid: "e3b9cbb8fe0945beb891b825281d5bfb"
format_stamp: "Formatted at 2026-09-28 15:00:08 on dist-test-slave-j968"
server_key: "3190af6063ecb0bbbd71d9775df3a410"
server_key_iv: "1e56583a12a5b4b0eb45352f18a991b5"
server_key_version: "encryptionkey@0"
I20260928 15:00:31.675067 15921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:00:31.721532 15921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:00:31.726234 15921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:00:31.726534 15921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:00:31.727659 15921 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:00:31.729307 15921 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:00:31.729442 15921 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:31.729578 15921 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:00:31.729674 15921 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:31.764655 15921 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.3:46849
I20260928 15:00:31.764750 16048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.3:46849 every 8 connection(s)
I20260928 15:00:31.766176 15921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
I20260928 15:00:31.770146 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 15921
Sep 28 15:00:31 dist-test-slave-j968 krb5kdc[15015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607631, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.3@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 15:00:31.787093 16053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:31.771204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33046 (local address 127.31.40.62:34035)
0928 15:00:31.771761 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:31.771766 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:31.772287 (+ 521us) server_negotiation.cc:407] Connection header received
0928 15:00:31.772447 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:31.772453 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:31.772541 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:31.772657 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:00:31.774665 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:31.775529 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:31.776975 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:31.777664 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:31.777753 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:31.781209 (+ 3456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:00:31.781215 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:00:31.781221 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:00:31.781258 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:00:31.784389 (+ 3131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:31.785081 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:31.785090 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:31.785093 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:31.785227 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:31.785623 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:31.785627 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:31.785629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:31.785893 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:00:31.786163 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:31.786459 (+ 296us) server_negotiation.cc:299] Negotiation successful
0928 15:00:31.786691 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":349,"threads_started":1}
I20260928 15:00:31.788501 15788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:31.768769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:57499 (local address 127.31.40.62:34035)
0928 15:00:31.769003 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:31.769010 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:31.770460 (+ 1450us) server_negotiation.cc:407] Connection header received
0928 15:00:31.773040 (+ 2580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:31.773046 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:31.773125 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:31.773234 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:00:31.774870 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:31.775701 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:31.777527 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:31.778437 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:31.778566 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:31.782284 (+ 3718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:00:31.782289 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:00:31.782292 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:00:31.782331 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:00:31.785181 (+ 2850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:31.785939 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:31.785943 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:31.785945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:31.786052 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:31.786489 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:31.786494 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:31.786497 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:31.786767 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:00:31.786940 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:31.787891 (+ 951us) server_negotiation.cc:299] Negotiation successful
0928 15:00:31.788131 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":81}
I20260928 15:00:31.789558 16051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:31.769286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34035 (local address 127.31.40.3:57499)
0928 15:00:31.770247 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:31.770314 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:31.772710 (+ 2396us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:31.773436 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:31.773465 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:31.774036 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:00:31.774694 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:31.774722 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:31.775911 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:31.775916 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:31.777368 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:31.777390 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:31.778668 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:31.778676 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:31.778990 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:31.779715 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:00:31.779748 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:00:31.782058 (+ 2310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:00:31.785334 (+ 3276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:31.785343 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:31.785360 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:31.785793 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:31.786167 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:31.786172 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:31.786175 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:31.786368 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:31.787044 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:00:31.787050 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:00:31.787504 (+ 454us) client_negotiation.cc:769] Sending connection context
0928 15:00:31.787780 (+ 276us) client_negotiation.cc:240] Negotiation successful
0928 15:00:31.788372 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":378,"threads_started":1}
I20260928 15:00:31.790825 16049 heartbeater.cc:347] Connected to a master server at 127.31.40.62:34035
I20260928 15:00:31.791097 16049 heartbeater.cc:464] Registering TS with master...
I20260928 15:00:31.791774 16049 heartbeater.cc:511] Master 127.31.40.62:34035 requested a full tablet report, sending...
I20260928 15:00:31.793146 15601 ts_manager.cc:194] Registered new tserver with Master: e3b9cbb8fe0945beb891b825281d5bfb (127.31.40.3:46849)
I20260928 15:00:31.793911 15601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.3@KRBTEST.COM'} at 127.31.40.3:57499
I20260928 15:00:31.802059 31904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:00:31.808660 15788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:31.802993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33054 (local address 127.31.40.62:34035)
0928 15:00:31.803253 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:31.803257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:31.803694 (+ 437us) server_negotiation.cc:407] Connection header received
0928 15:00:31.803920 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:31.803926 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:31.804076 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:31.804203 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:00:31.804612 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:31.805348 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:31.806868 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:31.807302 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:31.807395 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:31.807849 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:00:31.808169 (+ 320us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 15:00:31.808466 (+ 297us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33054: token expired
Metrics: {"server-negotiator.queue_time_us":154}
W20260928 15:00:31.808802 15788 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33054: token expired
W20260928 15:00:31.809347 15523 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.31.40.62:34035)
I20260928 15:00:31.821861 15788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:31.809873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33070 (local address 127.31.40.62:34035)
0928 15:00:31.810183 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:31.810187 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:31.810202 (+ 15us) server_negotiation.cc:407] Connection header received
0928 15:00:31.810239 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:31.810242 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:31.810296 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:31.810372 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:00:31.811772 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:31.812551 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:31.813860 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:31.814262 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:31.814355 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:31.817049 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:00:31.817054 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:00:31.817057 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:00:31.817089 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:00:31.819572 (+ 2483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:31.820292 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:31.820296 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:31.820298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:31.820411 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:31.820727 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:31.820730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:31.820732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:31.821054 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:00:31.821233 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:31.821456 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 15:00:31.821612 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173}
I20260928 15:00:31.825603 15601 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33070:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:00:31.833241 15601 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:00:31.834694 15786 heartbeater.cc:503] Master 127.31.40.62:34035 was elected leader, sending a full tablet report...
I20260928 15:00:31.879021 16062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:31.845583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60340 (local address 127.31.40.3:46849)
0928 15:00:31.869999 (+ 24416us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:31.870010 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:31.870059 (+ 49us) server_negotiation.cc:407] Connection header received
0928 15:00:31.870180 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:31.870188 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:31.870503 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:31.870745 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:00:31.872025 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:31.873753 (+ 1728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:31.876310 (+ 2557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:31.877544 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:31.877679 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:31.877941 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:31.878359 (+ 418us) server_negotiation.cc:299] Negotiation successful
0928 15:00:31.878561 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24194,"thread_start_us":24051,"threads_started":1}
I20260928 15:00:31.879930 16060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:31.844994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.3:46849 (local address 127.0.0.1:60340)
0928 15:00:31.860769 (+ 15775us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:31.860838 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:31.861123 (+ 285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:31.871136 (+ 10013us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:31.871143 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:31.871175 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:00:31.871757 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:31.871772 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:31.873920 (+ 2148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:31.873925 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:31.876140 (+ 2215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:31.876153 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:31.877767 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:31.877773 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:31.877967 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:31.878020 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 15:00:31.878149 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 15:00:31.879501 (+ 1352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15352,"mutex_wait_us":164,"thread_start_us":17457,"threads_started":1}
I20260928 15:00:31.880115 16061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:31.845512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:32837 (local address 127.0.0.1:36062)
0928 15:00:31.870777 (+ 25265us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:31.870798 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:31.870929 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:31.871648 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:31.871654 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:31.871684 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:00:31.872187 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:31.872199 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:31.873606 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:31.873613 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:31.876787 (+ 3174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:31.876802 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:31.879276 (+ 2474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:31.879282 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:31.879445 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:31.879481 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 15:00:31.879564 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 15:00:31.879693 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25134,"mutex_wait_us":337,"thread_start_us":25653,"threads_started":1}
I20260928 15:00:31.880223 16063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:31.845939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42112 (local address 127.31.40.2:32855)
0928 15:00:31.870393 (+ 24454us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:31.870404 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:31.870435 (+ 31us) server_negotiation.cc:407] Connection header received
0928 15:00:31.870536 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:31.870543 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:31.870788 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:31.871031 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:00:31.872257 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:31.873756 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:31.876982 (+ 3226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:31.878109 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:31.878221 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:31.878417 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:31.879654 (+ 1237us) server_negotiation.cc:299] Negotiation successful
0928 15:00:31.879824 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24277,"thread_start_us":24151,"threads_started":1}
I20260928 15:00:31.879532 16059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:31.845101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.2:32855 (local address 127.0.0.1:42112)
0928 15:00:31.862776 (+ 17675us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:31.862794 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:31.862910 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:31.871195 (+ 8285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:31.871200 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:31.871228 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:00:31.872032 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:31.872048 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:31.873922 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:31.873927 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:31.876787 (+ 2860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:31.876802 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:31.878623 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:31.878629 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:31.878814 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:31.878844 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 15:00:31.878925 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 15:00:31.879107 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17534,"mutex_wait_us":205,"thread_start_us":16328,"threads_started":1}
I20260928 15:00:31.880964 16064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:31.846167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36062 (local address 127.31.40.1:32837)
0928 15:00:31.869210 (+ 23043us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:31.869224 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:31.871063 (+ 1839us) server_negotiation.cc:407] Connection header received
0928 15:00:31.871166 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:31.871174 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:31.871413 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:31.871617 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:00:31.872361 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:31.873478 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:31.876956 (+ 3478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:31.878753 (+ 1797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:31.878874 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:31.879104 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:31.880504 (+ 1400us) server_negotiation.cc:299] Negotiation successful
0928 15:00:31.880673 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22785,"thread_start_us":23064,"threads_started":1}
I20260928 15:00:31.883626 15853 tablet_service.cc:1467] Processing CreateTablet for tablet e7633ef3dc114a838d4b88d1cc56a8a8 (DEFAULT_TABLE table=test-table [id=3aa3b3983dd14607be127f61bb52f3bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:00:31.884356 15984 tablet_service.cc:1467] Processing CreateTablet for tablet e7633ef3dc114a838d4b88d1cc56a8a8 (DEFAULT_TABLE table=test-table [id=3aa3b3983dd14607be127f61bb52f3bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:00:31.884356 15721 tablet_service.cc:1467] Processing CreateTablet for tablet e7633ef3dc114a838d4b88d1cc56a8a8 (DEFAULT_TABLE table=test-table [id=3aa3b3983dd14607be127f61bb52f3bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:00:31.885109 15853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7633ef3dc114a838d4b88d1cc56a8a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:00:31.885737 15984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7633ef3dc114a838d4b88d1cc56a8a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:00:31.885754 15721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7633ef3dc114a838d4b88d1cc56a8a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:00:31.896835 16065 tablet_bootstrap.cc:492] T e7633ef3dc114a838d4b88d1cc56a8a8 P 3559026efbc642419cabdda2898714a6: Bootstrap starting.
I20260928 15:00:31.900959 16067 tablet_bootstrap.cc:492] T e7633ef3dc114a838d4b88d1cc56a8a8 P e3b9cbb8fe0945beb891b825281d5bfb: Bootstrap starting.
I20260928 15:00:31.901087 16065 tablet_bootstrap.cc:654] T e7633ef3dc114a838d4b88d1cc56a8a8 P 3559026efbc642419cabdda2898714a6: Neither blocks nor log segments found. Creating new log.
I20260928 15:00:31.901829 16066 tablet_bootstrap.cc:492] T e7633ef3dc114a838d4b88d1cc56a8a8 P 14b53f4d618d4bfb96ce8b6fc0746dc0: Bootstrap starting.
I20260928 15:00:31.902439 16065 log.cc:824] T e7633ef3dc114a838d4b88d1cc56a8a8 P 3559026efbc642419cabdda2898714a6: Log is configured to *not* fsync() on all Append() calls
I20260928 15:00:31.905073 16065 tablet_bootstrap.cc:492] T e7633ef3dc114a838d4b88d1cc56a8a8 P 3559026efbc642419cabdda2898714a6: No bootstrap required, opened a new log
I20260928 15:00:31.905346 16065 ts_tablet_manager.cc:1397] T e7633ef3dc114a838d4b88d1cc56a8a8 P 3559026efbc642419cabdda2898714a6: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260928 15:00:31.905419 16066 tablet_bootstrap.cc:654] T e7633ef3dc114a838d4b88d1cc56a8a8 P 14b53f4d618d4bfb96ce8b6fc0746dc0: Neither blocks nor log segments found. Creating new log.
I20260928 15:00:31.906783 16066 log.cc:824] T e7633ef3dc114a838d4b88d1cc56a8a8 P 14b53f4d618d4bfb96ce8b6fc0746dc0: Log is configured to *not* fsync() on all Append() calls
I20260928 15:00:31.906883 16067 tablet_bootstrap.cc:654] T e7633ef3dc114a838d4b88d1cc56a8a8 P e3b9cbb8fe0945beb891b825281d5bfb: Neither blocks nor log segments found. Creating new log.
I20260928 15:00:31.908111 16067 log.cc:824] T e7633ef3dc114a838d4b88d1cc56a8a8 P e3b9cbb8fe0945beb891b825281d5bfb: Log is configured to *not* fsync() on all Append() calls
I20260928 15:00:31.909968 16066 tablet_bootstrap.cc:492] T e7633ef3dc114a838d4b88d1cc56a8a8 P 14b53f4d618d4bfb96ce8b6fc0746dc0: No bootstrap required, opened a new log
I20260928 15:00:31.910240 16066 ts_tablet_manager.cc:1397] T e7633ef3dc114a838d4b88d1cc56a8a8 P 14b53f4d618d4bfb96ce8b6fc0746dc0: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 15:00:31.911073 16067 tablet_bootstrap.cc:492] T e7633ef3dc114a838d4b88d1cc56a8a8 P e3b9cbb8fe0945beb891b825281d5bfb: No bootstrap required, opened a new log
I20260928 15:00:31.911324 16067 ts_tablet_manager.cc:1397] T e7633ef3dc114a838d4b88d1cc56a8a8 P e3b9cbb8fe0945beb891b825281d5bfb: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20260928 15:00:31.913872 16065 raft_consensus.cc:348] T e7633ef3dc114a838d4b88d1cc56a8a8 P 3559026efbc642419cabdda2898714a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b9cbb8fe0945beb891b825281d5bfb" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 46849 } } peers { permanent_uuid: "3559026efbc642419cabdda2898714a6" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 32855 } } peers { permanent_uuid: "14b53f4d618d4bfb96ce8b6fc0746dc0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 32837 } }
I20260928 15:00:31.914258 16065 raft_consensus.cc:374] T e7633ef3dc114a838d4b88d1cc56a8a8 P 3559026efbc642419cabdda2898714a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:00:31.914376 16065 raft_consensus.cc:729] T e7633ef3dc114a838d4b88d1cc56a8a8 P 3559026efbc642419cabdda2898714a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3559026efbc642419cabdda2898714a6, State: Initialized, Role: FOLLOWER
I20260928 15:00:31.915020 16065 consensus_queue.cc:260] T e7633ef3dc114a838d4b88d1cc56a8a8 P 3559026efbc642419cabdda2898714a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b9cbb8fe0945beb891b825281d5bfb" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 46849 } } peers { permanent_uuid: "3559026efbc642419cabdda2898714a6" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 32855 } } peers { permanent_uuid: "14b53f4d618d4bfb96ce8b6fc0746dc0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 32837 } }
I20260928 15:00:31.920614 15918 heartbeater.cc:503] Master 127.31.40.62:34035 was elected leader, sending a full tablet report...
I20260928 15:00:31.921190 16066 raft_consensus.cc:348] T e7633ef3dc114a838d4b88d1cc56a8a8 P 14b53f4d618d4bfb96ce8b6fc0746dc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b9cbb8fe0945beb891b825281d5bfb" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 46849 } } peers { permanent_uuid: "3559026efbc642419cabdda2898714a6" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 32855 } } peers { permanent_uuid: "14b53f4d618d4bfb96ce8b6fc0746dc0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 32837 } }
I20260928 15:00:31.921664 16066 raft_consensus.cc:374] T e7633ef3dc114a838d4b88d1cc56a8a8 P 14b53f4d618d4bfb96ce8b6fc0746dc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:00:31.921799 16066 raft_consensus.cc:729] T e7633ef3dc114a838d4b88d1cc56a8a8 P 14b53f4d618d4bfb96ce8b6fc0746dc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14b53f4d618d4bfb96ce8b6fc0746dc0, State: Initialized, Role: FOLLOWER
I20260928 15:00:31.921886 16067 raft_consensus.cc:348] T e7633ef3dc114a838d4b88d1cc56a8a8 P e3b9cbb8fe0945beb891b825281d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b9cbb8fe0945beb891b825281d5bfb" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 46849 } } peers { permanent_uuid: "3559026efbc642419cabdda2898714a6" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 32855 } } peers { permanent_uuid: "14b53f4d618d4bfb96ce8b6fc0746dc0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 32837 } }
I20260928 15:00:31.922308 16067 raft_consensus.cc:374] T e7633ef3dc114a838d4b88d1cc56a8a8 P e3b9cbb8fe0945beb891b825281d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:00:31.922405 16067 raft_consensus.cc:729] T e7633ef3dc114a838d4b88d1cc56a8a8 P e3b9cbb8fe0945beb891b825281d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3b9cbb8fe0945beb891b825281d5bfb, State: Initialized, Role: FOLLOWER
I20260928 15:00:31.922436 16066 consensus_queue.cc:260] T e7633ef3dc114a838d4b88d1cc56a8a8 P 14b53f4d618d4bfb96ce8b6fc0746dc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b9cbb8fe0945beb891b825281d5bfb" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 46849 } } peers { permanent_uuid: "3559026efbc642419cabdda2898714a6" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 32855 } } peers { permanent_uuid: "14b53f4d618d4bfb96ce8b6fc0746dc0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 32837 } }
I20260928 15:00:31.922984 16065 ts_tablet_manager.cc:1428] T e7633ef3dc114a838d4b88d1cc56a8a8 P 3559026efbc642419cabdda2898714a6: Time spent starting tablet: real 0.018s user 0.010s sys 0.007s
I20260928 15:00:31.923137 16067 consensus_queue.cc:260] T e7633ef3dc114a838d4b88d1cc56a8a8 P e3b9cbb8fe0945beb891b825281d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b9cbb8fe0945beb891b825281d5bfb" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 46849 } } peers { permanent_uuid: "3559026efbc642419cabdda2898714a6" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 32855 } } peers { permanent_uuid: "14b53f4d618d4bfb96ce8b6fc0746dc0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 32837 } }
I20260928 15:00:31.924817 16066 ts_tablet_manager.cc:1428] T e7633ef3dc114a838d4b88d1cc56a8a8 P 14b53f4d618d4bfb96ce8b6fc0746dc0: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
I20260928 15:00:31.925666 16049 heartbeater.cc:503] Master 127.31.40.62:34035 was elected leader, sending a full tablet report...
I20260928 15:00:31.926496 16067 ts_tablet_manager.cc:1428] T e7633ef3dc114a838d4b88d1cc56a8a8 P e3b9cbb8fe0945beb891b825281d5bfb: Time spent starting tablet: real 0.015s user 0.015s sys 0.001s
W20260928 15:00:32.017939 16050 tablet.cc:2369] T e7633ef3dc114a838d4b88d1cc56a8a8 P e3b9cbb8fe0945beb891b825281d5bfb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:00:32.026196 15919 tablet.cc:2369] T e7633ef3dc114a838d4b88d1cc56a8a8 P 3559026efbc642419cabdda2898714a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:00:32.049191 15787 tablet.cc:2369] T e7633ef3dc114a838d4b88d1cc56a8a8 P 14b53f4d618d4bfb96ce8b6fc0746dc0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:00:32.050261 16072 raft_consensus.cc:482] T e7633ef3dc114a838d4b88d1cc56a8a8 P 14b53f4d618d4bfb96ce8b6fc0746dc0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:00:32.050499 16072 raft_consensus.cc:504] T e7633ef3dc114a838d4b88d1cc56a8a8 P 14b53f4d618d4bfb96ce8b6fc0746dc0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b9cbb8fe0945beb891b825281d5bfb" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 46849 } } peers { permanent_uuid: "3559026efbc642419cabdda2898714a6" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 32855 } } peers { permanent_uuid: "14b53f4d618d4bfb96ce8b6fc0746dc0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 32837 } }
I20260928 15:00:32.052309 16072 leader_election.cc:288] T e7633ef3dc114a838d4b88d1cc56a8a8 P 14b53f4d618d4bfb96ce8b6fc0746dc0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3b9cbb8fe0945beb891b825281d5bfb (127.31.40.3:46849), 3559026efbc642419cabdda2898714a6 (127.31.40.2:32855)
I20260928 15:00:32.061386 16062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:32.052714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:42065 (local address 127.31.40.3:46849)
0928 15:00:32.052932 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:32.052937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:32.053620 (+ 683us) server_negotiation.cc:407] Connection header received
0928 15:00:32.053839 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:32.053845 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:32.053952 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:32.054040 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:00:32.054753 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:32.055900 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:32.059049 (+ 3149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:32.060338 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:32.060594 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:32.060708 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:32.060974 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 15:00:32.061112 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 15:00:32.061415 16063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:32.052937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:46297 (local address 127.31.40.2:32855)
0928 15:00:32.053169 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:32.053177 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:32.053767 (+ 590us) server_negotiation.cc:407] Connection header received
0928 15:00:32.053854 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:32.053859 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:32.053932 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:32.054041 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:00:32.054753 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:32.055902 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:32.059049 (+ 3147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:32.060336 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:32.060635 (+ 299us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:32.060725 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:32.061021 (+ 296us) server_negotiation.cc:299] Negotiation successful
0928 15:00:32.061149 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 15:00:32.061553 16074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:32.052663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.3:46849 (local address 127.31.40.1:42065)
0928 15:00:32.053493 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:32.053515 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:32.053683 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:32.054072 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:32.054077 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:32.054169 (+ 92us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:00:32.054604 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:32.054613 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:32.056047 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:32.056052 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:32.058872 (+ 2820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:32.058889 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:32.060557 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:32.060561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:32.060769 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:32.060806 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 15:00:32.060899 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 15:00:32.061047 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"mutex_wait_us":243,"spinlock_wait_cycles":23424,"thread_start_us":439,"threads_started":1}
I20260928 15:00:32.061975 16075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:32.052722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.2:32855 (local address 127.31.40.1:46297)
0928 15:00:32.053642 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:32.053657 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:32.053764 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:32.054072 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:32.054077 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:32.054105 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:00:32.054593 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:32.054605 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:32.056048 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:32.056052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:32.058872 (+ 2820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:32.058889 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:32.060701 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:32.060707 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:32.060861 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:32.060899 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 15:00:32.060983 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 15:00:32.061081 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":815,"mutex_wait_us":165,"thread_start_us":622,"threads_started":1}
I20260928 15:00:32.062642 16004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7633ef3dc114a838d4b88d1cc56a8a8" candidate_uuid: "14b53f4d618d4bfb96ce8b6fc0746dc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3b9cbb8fe0945beb891b825281d5bfb" is_pre_election: true
I20260928 15:00:32.062943 15873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7633ef3dc114a838d4b88d1cc56a8a8" candidate_uuid: "14b53f4d618d4bfb96ce8b6fc0746dc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3559026efbc642419cabdda2898714a6" is_pre_election: true
I20260928 15:00:32.063199 16004 raft_consensus.cc:2454] T e7633ef3dc114a838d4b88d1cc56a8a8 P e3b9cbb8fe0945beb891b825281d5bfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14b53f4d618d4bfb96ce8b6fc0746dc0 in term 0.
I20260928 15:00:32.063432 15873 raft_consensus.cc:2454] T e7633ef3dc114a838d4b88d1cc56a8a8 P 3559026efbc642419cabdda2898714a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14b53f4d618d4bfb96ce8b6fc0746dc0 in term 0.
I20260928 15:00:32.063990 15674 leader_election.cc:302] T e7633ef3dc114a838d4b88d1cc56a8a8 P 14b53f4d618d4bfb96ce8b6fc0746dc0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14b53f4d618d4bfb96ce8b6fc0746dc0, e3b9cbb8fe0945beb891b825281d5bfb; no voters:
I20260928 15:00:32.064440 16072 raft_consensus.cc:2781] T e7633ef3dc114a838d4b88d1cc56a8a8 P 14b53f4d618d4bfb96ce8b6fc0746dc0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:00:32.064606 16072 raft_consensus.cc:482] T e7633ef3dc114a838d4b88d1cc56a8a8 P 14b53f4d618d4bfb96ce8b6fc0746dc0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:00:32.064730 16072 raft_consensus.cc:3037] T e7633ef3dc114a838d4b88d1cc56a8a8 P 14b53f4d618d4bfb96ce8b6fc0746dc0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:00:32.067005 16072 raft_consensus.cc:504] T e7633ef3dc114a838d4b88d1cc56a8a8 P 14b53f4d618d4bfb96ce8b6fc0746dc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b9cbb8fe0945beb891b825281d5bfb" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 46849 } } peers { permanent_uuid: "3559026efbc642419cabdda2898714a6" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 32855 } } peers { permanent_uuid: "14b53f4d618d4bfb96ce8b6fc0746dc0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 32837 } }
I20260928 15:00:32.067670 16072 leader_election.cc:288] T e7633ef3dc114a838d4b88d1cc56a8a8 P 14b53f4d618d4bfb96ce8b6fc0746dc0 [CANDIDATE]: Term 1 election: Requested vote from peers e3b9cbb8fe0945beb891b825281d5bfb (127.31.40.3:46849), 3559026efbc642419cabdda2898714a6 (127.31.40.2:32855)
I20260928 15:00:32.068238 15873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7633ef3dc114a838d4b88d1cc56a8a8" candidate_uuid: "14b53f4d618d4bfb96ce8b6fc0746dc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3559026efbc642419cabdda2898714a6"
I20260928 15:00:32.068292 16004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7633ef3dc114a838d4b88d1cc56a8a8" candidate_uuid: "14b53f4d618d4bfb96ce8b6fc0746dc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3b9cbb8fe0945beb891b825281d5bfb"
I20260928 15:00:32.068495 15873 raft_consensus.cc:3037] T e7633ef3dc114a838d4b88d1cc56a8a8 P 3559026efbc642419cabdda2898714a6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:00:32.068524 16004 raft_consensus.cc:3037] T e7633ef3dc114a838d4b88d1cc56a8a8 P e3b9cbb8fe0945beb891b825281d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:00:32.070704 15873 raft_consensus.cc:2454] T e7633ef3dc114a838d4b88d1cc56a8a8 P 3559026efbc642419cabdda2898714a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14b53f4d618d4bfb96ce8b6fc0746dc0 in term 1.
I20260928 15:00:32.070729 16004 raft_consensus.cc:2454] T e7633ef3dc114a838d4b88d1cc56a8a8 P e3b9cbb8fe0945beb891b825281d5bfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14b53f4d618d4bfb96ce8b6fc0746dc0 in term 1.
I20260928 15:00:32.071190 15677 leader_election.cc:302] T e7633ef3dc114a838d4b88d1cc56a8a8 P 14b53f4d618d4bfb96ce8b6fc0746dc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14b53f4d618d4bfb96ce8b6fc0746dc0, 3559026efbc642419cabdda2898714a6; no voters:
I20260928 15:00:32.071522 16072 raft_consensus.cc:2781] T e7633ef3dc114a838d4b88d1cc56a8a8 P 14b53f4d618d4bfb96ce8b6fc0746dc0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:00:32.072017 16072 raft_consensus.cc:686] T e7633ef3dc114a838d4b88d1cc56a8a8 P 14b53f4d618d4bfb96ce8b6fc0746dc0 [term 1 LEADER]: Becoming Leader. State: Replica: 14b53f4d618d4bfb96ce8b6fc0746dc0, State: Running, Role: LEADER
I20260928 15:00:32.072511 16072 consensus_queue.cc:237] T e7633ef3dc114a838d4b88d1cc56a8a8 P 14b53f4d618d4bfb96ce8b6fc0746dc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b9cbb8fe0945beb891b825281d5bfb" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 46849 } } peers { permanent_uuid: "3559026efbc642419cabdda2898714a6" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 32855 } } peers { permanent_uuid: "14b53f4d618d4bfb96ce8b6fc0746dc0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 32837 } }
I20260928 15:00:32.077136 15601 catalog_manager.cc:5495] T e7633ef3dc114a838d4b88d1cc56a8a8 P 14b53f4d618d4bfb96ce8b6fc0746dc0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 14b53f4d618d4bfb96ce8b6fc0746dc0 (127.31.40.1). New cstate: current_term: 1 leader_uuid: "14b53f4d618d4bfb96ce8b6fc0746dc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b9cbb8fe0945beb891b825281d5bfb" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 46849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3559026efbc642419cabdda2898714a6" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 32855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14b53f4d618d4bfb96ce8b6fc0746dc0" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 32837 } health_report { overall_health: HEALTHY } } }
I20260928 15:00:32.115934 16064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:32.110140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36072 (local address 127.31.40.1:32837)
0928 15:00:32.110356 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:32.110362 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:32.110483 (+ 121us) server_negotiation.cc:407] Connection header received
0928 15:00:32.110737 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:32.110741 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:32.110849 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:32.111027 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:00:32.111419 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:32.112251 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:32.113646 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:32.114075 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:32.114209 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:32.114751 (+ 542us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:00:32.115277 (+ 526us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:00:32.115440 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:32.115520 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 15:00:32.115602 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 15:00:32.124521 15873 raft_consensus.cc:1261] T e7633ef3dc114a838d4b88d1cc56a8a8 P 3559026efbc642419cabdda2898714a6 [term 1 FOLLOWER]: Refusing update from remote peer 14b53f4d618d4bfb96ce8b6fc0746dc0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:00:32.124534 16004 raft_consensus.cc:1261] T e7633ef3dc114a838d4b88d1cc56a8a8 P e3b9cbb8fe0945beb891b825281d5bfb [term 1 FOLLOWER]: Refusing update from remote peer 14b53f4d618d4bfb96ce8b6fc0746dc0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:00:32.125540 16076 consensus_queue.cc:1035] T e7633ef3dc114a838d4b88d1cc56a8a8 P 14b53f4d618d4bfb96ce8b6fc0746dc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3559026efbc642419cabdda2898714a6" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 32855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:00:32.125900 16072 consensus_queue.cc:1035] T e7633ef3dc114a838d4b88d1cc56a8a8 P 14b53f4d618d4bfb96ce8b6fc0746dc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b9cbb8fe0945beb891b825281d5bfb" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 46849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:00:32.158317 15601 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:00:32.158612 15601 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:00:32.162207 15601 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69: Sending DeleteTablet for 3 replicas of tablet e7633ef3dc114a838d4b88d1cc56a8a8
I20260928 15:00:32.163312 15984 tablet_service.cc:1514] Processing DeleteTablet for tablet e7633ef3dc114a838d4b88d1cc56a8a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:00:32 UTC) from {username='kudu', principal='kudu/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:60340
I20260928 15:00:32.163456 15721 tablet_service.cc:1514] Processing DeleteTablet for tablet e7633ef3dc114a838d4b88d1cc56a8a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:00:32 UTC) from {username='kudu', principal='kudu/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:36062
I20260928 15:00:32.164022 15853 tablet_service.cc:1514] Processing DeleteTablet for tablet e7633ef3dc114a838d4b88d1cc56a8a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:00:32 UTC) from {username='kudu', principal='kudu/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:42112
I20260928 15:00:32.165032 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 15638
I20260928 15:00:32.179175 16090 tablet_replica.cc:330] T e7633ef3dc114a838d4b88d1cc56a8a8 P 3559026efbc642419cabdda2898714a6: stopping tablet replica
I20260928 15:00:32.179731 16090 raft_consensus.cc:2229] T e7633ef3dc114a838d4b88d1cc56a8a8 P 3559026efbc642419cabdda2898714a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:00:32.180083 16089 tablet_replica.cc:330] T e7633ef3dc114a838d4b88d1cc56a8a8 P 14b53f4d618d4bfb96ce8b6fc0746dc0: stopping tablet replica
I20260928 15:00:32.180534 16089 raft_consensus.cc:2229] T e7633ef3dc114a838d4b88d1cc56a8a8 P 14b53f4d618d4bfb96ce8b6fc0746dc0 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:00:32.180794 16090 raft_consensus.cc:2258] T e7633ef3dc114a838d4b88d1cc56a8a8 P 3559026efbc642419cabdda2898714a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:00:32.181282 16089 raft_consensus.cc:2258] T e7633ef3dc114a838d4b88d1cc56a8a8 P 14b53f4d618d4bfb96ce8b6fc0746dc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:00:32.182149 16088 tablet_replica.cc:330] T e7633ef3dc114a838d4b88d1cc56a8a8 P e3b9cbb8fe0945beb891b825281d5bfb: stopping tablet replica
I20260928 15:00:32.182581 16088 raft_consensus.cc:2229] T e7633ef3dc114a838d4b88d1cc56a8a8 P e3b9cbb8fe0945beb891b825281d5bfb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:00:32.183089 16088 raft_consensus.cc:2258] T e7633ef3dc114a838d4b88d1cc56a8a8 P e3b9cbb8fe0945beb891b825281d5bfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:00:32.183676 16089 ts_tablet_manager.cc:1905] T e7633ef3dc114a838d4b88d1cc56a8a8 P 14b53f4d618d4bfb96ce8b6fc0746dc0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:00:32.185087 16090 ts_tablet_manager.cc:1905] T e7633ef3dc114a838d4b88d1cc56a8a8 P 3559026efbc642419cabdda2898714a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:00:32.191612 16089 ts_tablet_manager.cc:1918] T e7633ef3dc114a838d4b88d1cc56a8a8 P 14b53f4d618d4bfb96ce8b6fc0746dc0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:00:32.191728 16064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:32.166898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36084 (local address 127.31.40.1:32837)
0928 15:00:32.167128 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:32.167136 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:32.167164 (+ 28us) server_negotiation.cc:407] Connection header received
0928 15:00:32.167232 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:32.167239 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:32.167334 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:32.167464 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:00:32.169972 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:32.171683 (+ 1711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:32.175495 (+ 3812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:32.176491 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:32.176648 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:32.182242 (+ 5594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:00:32.182258 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:00:32.182278 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:00:32.182329 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:00:32.187052 (+ 4723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:32.188084 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:32.188094 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:32.188097 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:32.188282 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:32.188791 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:32.188796 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:32.188799 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:32.189308 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:00:32.189629 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:32.190917 (+ 1288us) server_negotiation.cc:299] Negotiation successful
0928 15:00:32.191196 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 15:00:32.191812 16089 log.cc:1196] T e7633ef3dc114a838d4b88d1cc56a8a8 P 14b53f4d618d4bfb96ce8b6fc0746dc0: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-0/wal/wals/e7633ef3dc114a838d4b88d1cc56a8a8
I20260928 15:00:32.192109 16090 ts_tablet_manager.cc:1918] T e7633ef3dc114a838d4b88d1cc56a8a8 P 3559026efbc642419cabdda2898714a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:00:32.192288 16090 log.cc:1196] T e7633ef3dc114a838d4b88d1cc56a8a8 P 3559026efbc642419cabdda2898714a6: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-1/wal/wals/e7633ef3dc114a838d4b88d1cc56a8a8
I20260928 15:00:32.192492 16089 ts_tablet_manager.cc:1939] T e7633ef3dc114a838d4b88d1cc56a8a8 P 14b53f4d618d4bfb96ce8b6fc0746dc0: Deleting consensus metadata
I20260928 15:00:32.192556 15781 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:00:32.192863 16090 ts_tablet_manager.cc:1939] T e7633ef3dc114a838d4b88d1cc56a8a8 P 3559026efbc642419cabdda2898714a6: Deleting consensus metadata
I20260928 15:00:32.194622 15588 catalog_manager.cc:4841] TS 3559026efbc642419cabdda2898714a6 (127.31.40.2:32855): tablet e7633ef3dc114a838d4b88d1cc56a8a8 (table test-table [id=3aa3b3983dd14607be127f61bb52f3bb]) successfully deleted
I20260928 15:00:32.196370 16088 ts_tablet_manager.cc:1905] T e7633ef3dc114a838d4b88d1cc56a8a8 P e3b9cbb8fe0945beb891b825281d5bfb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:00:32.204438 16088 ts_tablet_manager.cc:1918] T e7633ef3dc114a838d4b88d1cc56a8a8 P e3b9cbb8fe0945beb891b825281d5bfb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:00:32.204649 16088 log.cc:1196] T e7633ef3dc114a838d4b88d1cc56a8a8 P e3b9cbb8fe0945beb891b825281d5bfb: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-2/wal/wals/e7633ef3dc114a838d4b88d1cc56a8a8
I20260928 15:00:32.205291 16088 ts_tablet_manager.cc:1939] T e7633ef3dc114a838d4b88d1cc56a8a8 P e3b9cbb8fe0945beb891b825281d5bfb: Deleting consensus metadata
I20260928 15:00:32.207284 15585 catalog_manager.cc:4841] TS e3b9cbb8fe0945beb891b825281d5bfb (127.31.40.3:46849): tablet e7633ef3dc114a838d4b88d1cc56a8a8 (table test-table [id=3aa3b3983dd14607be127f61bb52f3bb]) successfully deleted
I20260928 15:00:32.301478 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 15638
I20260928 15:00:32.301522 15586 catalog_manager.cc:4841] TS 14b53f4d618d4bfb96ce8b6fc0746dc0 (127.31.40.1:32837): tablet e7633ef3dc114a838d4b88d1cc56a8a8 (table test-table [id=3aa3b3983dd14607be127f61bb52f3bb]) successfully deleted
I20260928 15:00:32.321048 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 15790
I20260928 15:00:32.334735 16063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:32.321813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42124 (local address 127.31.40.2:32855)
0928 15:00:32.321999 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:32.322004 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:32.322091 (+ 87us) server_negotiation.cc:407] Connection header received
0928 15:00:32.322255 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:32.322260 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:32.322409 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:32.322523 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:00:32.323789 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:32.324583 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:32.325868 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:32.326248 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:32.326340 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:32.329209 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:00:32.329223 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:00:32.329241 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:00:32.329279 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:00:32.332319 (+ 3040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:32.332980 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:32.332988 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:32.332990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:32.333107 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:32.333420 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:32.333423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:32.333425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:32.333797 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:00:32.334030 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:32.334296 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 15:00:32.334477 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 15:00:32.335408 15913 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:00:32.427277 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 15790
I20260928 15:00:32.445640 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 15921
I20260928 15:00:32.459798 16062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:32.446478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60352 (local address 127.31.40.3:46849)
0928 15:00:32.446658 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:32.446663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:32.446682 (+ 19us) server_negotiation.cc:407] Connection header received
0928 15:00:32.446791 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:32.446795 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:32.446872 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:32.446957 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:00:32.448420 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:32.449169 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:32.450461 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:32.450866 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:32.450971 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:32.453926 (+ 2955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:00:32.453942 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:00:32.453960 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:00:32.454004 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:00:32.456986 (+ 2982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:32.457727 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:32.457734 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:32.457737 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:32.457882 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:32.458249 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:32.458253 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:32.458255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:32.458623 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:00:32.458842 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:32.459220 (+ 378us) server_negotiation.cc:299] Negotiation successful
0928 15:00:32.459390 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 15:00:32.460547 16044 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:00:32.550230 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 15921
I20260928 15:00:32.568049 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 15568
I20260928 15:00:32.569260 15631 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:00:32.685346 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 15568
I20260928 15:00:52.708840 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:34035
--webserver_interface=127.31.40.62
--webserver_port=36777
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:34399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.40.62:34035
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:00:53.048161 16095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:00:53.048504 16095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:00:53.048653 16095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:00:53.056206 16095 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 15:00:53.056335 16095 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 15:00:53.056421 16095 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 15:00:53.056512 16095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:00:53.056608 16095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:00:53.056720 16095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:00:53.056810 16095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:00:53.056892 16095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:00:53.067011 16095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:34399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.31.40.62:34035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:34035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.40.62
--webserver_port=36777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:00:53.068480 16095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:00:53.069831 16095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:00:53.082329 16103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:00:53.082590 16095 server_base.cc:1034] running on GCE node
W20260928 15:00:53.082391 16100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:53.082355 16101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:00:53.083860 16095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:00:53.085306 16095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:00:53.086552 16095 hybrid_clock.cc:648] HybridClock initialized: now 1790607653086475 us; error 39 us; skew 500 ppm
I20260928 15:00:53.086874 16095 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:00:53 dist-test-slave-j968 krb5kdc[15015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607653, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:00:53.090010 16095 init.cc:377] Logged in from keytab as kudu/127.31.40.62@KRBTEST.COM (short username kudu)
I20260928 15:00:53.091471 16095 webserver.cc:460] Webserver started at http://127.31.40.62:36777/ using document root <none> and password file <none>
I20260928 15:00:53.092222 16095 fs_manager.cc:359] Metadata directory not provided
I20260928 15:00:53.092316 16095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:00:53.097797 16095 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 15:00:53.100594 16110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:53.101907 16095 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:00:53.102041 16095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/master-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/master-0/wal
uuid: "99120398a8b0443aa9313e4874f4aa69"
format_stamp: "Formatted at 2026-09-28 15:00:06 on dist-test-slave-j968"
server_key: "c032cd757ec4b7ee1e8e39128856c2ea"
server_key_iv: "84998d5837221dcbb3e30753c62ad036"
server_key_version: "encryptionkey@0"
I20260928 15:00:53.102988 16095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:00:53.126806 16095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:00:53.131394 16095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:00:53.131716 16095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:00:53.151119 16095 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.62:34035
I20260928 15:00:53.151142 16161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.62:34035 every 8 connection(s)
I20260928 15:00:53.152817 16095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
I20260928 15:00:53.153736 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 16095
I20260928 15:00:53.154814 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.1:32837
--local_ip_for_outbound_sockets=127.31.40.1
--tserver_master_addrs=127.31.40.62:34035
--webserver_port=38983
--webserver_interface=127.31.40.1
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab
--principal=kudu/127.31.40.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:34399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:00:53.163450 16162 sys_catalog.cc:263] Verifying existing consensus state
I20260928 15:00:53.167008 16162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69: Bootstrap starting.
I20260928 15:00:53.188073 16162 log.cc:824] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69: Log is configured to *not* fsync() on all Append() calls
I20260928 15:00:53.204032 16162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 15:00:53.204704 16162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69: Bootstrap complete.
I20260928 15:00:53.214111 16162 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99120398a8b0443aa9313e4874f4aa69" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34035 } }
I20260928 15:00:53.215229 16162 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99120398a8b0443aa9313e4874f4aa69, State: Initialized, Role: FOLLOWER
I20260928 15:00:53.215989 16162 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99120398a8b0443aa9313e4874f4aa69" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34035 } }
I20260928 15:00:53.216238 16162 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:00:53.216359 16162 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:00:53.216552 16162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69 [term 2 FOLLOWER]: Advancing to term 3
I20260928 15:00:53.220500 16162 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99120398a8b0443aa9313e4874f4aa69" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34035 } }
I20260928 15:00:53.221004 16162 leader_election.cc:302] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99120398a8b0443aa9313e4874f4aa69; no voters:
I20260928 15:00:53.221658 16162 leader_election.cc:288] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 15:00:53.221879 16166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 15:00:53.223083 16166 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69 [term 3 LEADER]: Becoming Leader. State: Replica: 99120398a8b0443aa9313e4874f4aa69, State: Running, Role: LEADER
I20260928 15:00:53.223645 16166 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99120398a8b0443aa9313e4874f4aa69" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34035 } }
I20260928 15:00:53.223839 16162 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:00:53.226835 16168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99120398a8b0443aa9313e4874f4aa69. Latest consensus state: current_term: 3 leader_uuid: "99120398a8b0443aa9313e4874f4aa69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99120398a8b0443aa9313e4874f4aa69" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34035 } } }
I20260928 15:00:53.227285 16168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69 [sys.catalog]: This master's current role is: LEADER
I20260928 15:00:53.227981 16167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "99120398a8b0443aa9313e4874f4aa69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99120398a8b0443aa9313e4874f4aa69" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34035 } } }
I20260928 15:00:53.228359 16167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69 [sys.catalog]: This master's current role is: LEADER
I20260928 15:00:53.237480 16174 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:00:53.248646 16174 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:00:53.250742 16174 catalog_manager.cc:1259] Loaded cluster ID: bfc018e2a8b7437587bf5b8abf7d1632
I20260928 15:00:53.250955 16174 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:00:53.256862 16174 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:00:53.259029 16174 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69: Loaded TSK: 0
I20260928 15:00:53.259315 16174 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 99120398a8b0443aa9313e4874f4aa69: Loaded TSK: 1
F20260928 15:00:53.260907 16174 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f97a63399ad google::LogMessage::Fail() at ??:0
I20260928 15:00:53.265352 16183 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f97a633c88a google::LogMessage::SendToLog() at ??:0
@ 0x7f97a63394de google::LogMessage::Flush() at ??:0
@ 0x7f97a633aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f97aa09d7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f97aa08c73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f97bbd681c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f97bbd67783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f97bbdf32a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f97bbdf30c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f97bbe3d5e9 std::function<>::operator()() at ??:0
@ 0x7f97bbd6bcdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f97bbd69c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f97bbdf1610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f97bbdf1439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f97a8f85345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f97a8f8891c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f97a8f88799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f97a8f63eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f97ac23d6db start_thread at ??:0
@ 0x7f97a51d361f clone at ??:0
W20260928 15:00:53.565538 16164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:00:53.565843 16164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:00:53.566007 16164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:00:53.573544 16164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:00:53.573661 16164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:00:53.573880 16164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.1
I20260928 15:00:53.584134 16164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:34399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab
--principal=kudu/127.31.40.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.1:32837
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.40.1
--webserver_port=38983
--tserver_master_addrs=127.31.40.62:34035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.1
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:00:53.585475 16164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:00:53.586788 16164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:00:53.600571 16209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:53.600484 16212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:53.600509 16210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:00:53.601756 16164 server_base.cc:1034] running on GCE node
I20260928 15:00:53.602420 16164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:00:53.603441 16164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:00:53.604640 16164 hybrid_clock.cc:648] HybridClock initialized: now 1790607653604591 us; error 35 us; skew 500 ppm
I20260928 15:00:53.605017 16164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:00:53 dist-test-slave-j968 krb5kdc[15015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607653, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:00:53.608250 16164 init.cc:377] Logged in from keytab as kudu/127.31.40.1@KRBTEST.COM (short username kudu)
I20260928 15:00:53.610124 16164 webserver.cc:460] Webserver started at http://127.31.40.1:38983/ using document root <none> and password file <none>
I20260928 15:00:53.610837 16164 fs_manager.cc:359] Metadata directory not provided
I20260928 15:00:53.610982 16164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:00:53.616396 16164 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:00:53.619164 16219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:53.620514 16164 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:00:53.620698 16164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-0/wal
uuid: "14b53f4d618d4bfb96ce8b6fc0746dc0"
format_stamp: "Formatted at 2026-09-28 15:00:07 on dist-test-slave-j968"
server_key: "ee4d4c5d194c7304fecefb3eef4fd5e2"
server_key_iv: "f6b65b0ead875215fb8c94b737eab460"
server_key_version: "encryptionkey@0"
I20260928 15:00:53.621688 16164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:00:53.676067 16164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:00:53.681203 16164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:00:53.681547 16164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:00:53.682700 16164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:00:53.684291 16164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:00:53.684441 16164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:53.684567 16164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:00:53.684669 16164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:53.721168 16164 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.1:32837
I20260928 15:00:53.721256 16331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.1:32837 every 8 connection(s)
I20260928 15:00:53.722823 16164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
I20260928 15:00:53.724769 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 16164
I20260928 15:00:53.725618 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.2:32855
--local_ip_for_outbound_sockets=127.31.40.2
--tserver_master_addrs=127.31.40.62:34035
--webserver_port=36365
--webserver_interface=127.31.40.2
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab
--principal=kudu/127.31.40.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:34399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:00:53.728623 16334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:53.726418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34035 (local address 127.31.40.1:40087)
0928 15:00:53.727282 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:53.727511 (+ 229us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":395,"threads_started":1}
W20260928 15:00:53.729480 16332 heartbeater.cc:644] Failed to heartbeat to 127.31.40.62:34035 (0 consecutive failures): Network error: Failed to ping master at 127.31.40.62:34035: Client connection negotiation failed: client connection to 127.31.40.62:34035: connect: Connection refused (error 111)
I20260928 15:00:53.730458 16334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:53.730087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34035 (local address 127.31.40.1:49621)
0928 15:00:53.730250 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:53.730331 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 15:00:53.731470 16334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:53.731137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34035 (local address 127.31.40.1:60959)
0928 15:00:53.731268 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:53.731342 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 15:00:53.731760 16332 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 15:00:54.062845 16335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:00:54.063119 16335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:00:54.063292 16335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:00:54.070510 16335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:00:54.070621 16335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:00:54.070827 16335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.2
I20260928 15:00:54.080564 16335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:34399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab
--principal=kudu/127.31.40.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.2:32855
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.40.2
--webserver_port=36365
--tserver_master_addrs=127.31.40.62:34035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.2
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:00:54.081903 16335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:00:54.083218 16335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:00:54.096453 16340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:54.096510 16341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:54.096482 16343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:00:54.097244 16335 server_base.cc:1034] running on GCE node
I20260928 15:00:54.097880 16335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:00:54.098855 16335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:00:54.100070 16335 hybrid_clock.cc:648] HybridClock initialized: now 1790607654100018 us; error 44 us; skew 500 ppm
I20260928 15:00:54.100396 16335 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:00:54 dist-test-slave-j968 krb5kdc[15015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607654, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:00:54.103524 16335 init.cc:377] Logged in from keytab as kudu/127.31.40.2@KRBTEST.COM (short username kudu)
I20260928 15:00:54.105080 16335 webserver.cc:460] Webserver started at http://127.31.40.2:36365/ using document root <none> and password file <none>
I20260928 15:00:54.105741 16335 fs_manager.cc:359] Metadata directory not provided
I20260928 15:00:54.105866 16335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:00:54.111478 16335 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260928 15:00:54.114331 16350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:54.115698 16335 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 15:00:54.115952 16335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-1/wal
uuid: "3559026efbc642419cabdda2898714a6"
format_stamp: "Formatted at 2026-09-28 15:00:08 on dist-test-slave-j968"
server_key: "bcb92fd637f53804a8d6c6b0460b1226"
server_key_iv: "3c0fd12e9261b1e47506f8167fe68de4"
server_key_version: "encryptionkey@0"
I20260928 15:00:54.116973 16335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:00:54.132331 16335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:00:54.138238 16335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:00:54.138568 16335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:00:54.139983 16335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:00:54.141800 16335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:00:54.141896 16335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:54.142031 16335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:00:54.142127 16335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:54.175133 16335 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.2:32855
I20260928 15:00:54.175215 16462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.2:32855 every 8 connection(s)
I20260928 15:00:54.176623 16335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
I20260928 15:00:54.181775 16465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:54.179751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34035 (local address 127.31.40.2:50357)
0928 15:00:54.180527 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:54.180734 (+ 207us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":275,"threads_started":1}
I20260928 15:00:54.181915 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 16335
W20260928 15:00:54.182530 16463 heartbeater.cc:644] Failed to heartbeat to 127.31.40.62:34035 (0 consecutive failures): Network error: Failed to ping master at 127.31.40.62:34035: Client connection negotiation failed: client connection to 127.31.40.62:34035: connect: Connection refused (error 111)
I20260928 15:00:54.182603 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.3:46849
--local_ip_for_outbound_sockets=127.31.40.3
--tserver_master_addrs=127.31.40.62:34035
--webserver_port=33821
--webserver_interface=127.31.40.3
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab
--principal=kudu/127.31.40.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:34399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:00:54.183522 16465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:54.183063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34035 (local address 127.31.40.2:60013)
0928 15:00:54.183253 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:54.183350 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 15:00:54.184510 16465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:54.184084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34035 (local address 127.31.40.2:35733)
0928 15:00:54.184248 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:54.184343 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 15:00:54.184770 16463 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 15:00:54.522646 16466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:00:54.522946 16466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:00:54.523043 16466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:00:54.530606 16466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:00:54.530723 16466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:00:54.530851 16466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.3
I20260928 15:00:54.540854 16466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:34399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab
--principal=kudu/127.31.40.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.3:46849
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.40.3
--webserver_port=33821
--tserver_master_addrs=127.31.40.62:34035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.3
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:00:54.542162 16466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:00:54.543432 16466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:00:54.557610 16471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:54.557682 16472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:54.557708 16474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:00:54.558516 16466 server_base.cc:1034] running on GCE node
I20260928 15:00:54.559108 16466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:00:54.560113 16466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:00:54.561312 16466 hybrid_clock.cc:648] HybridClock initialized: now 1790607654561263 us; error 37 us; skew 500 ppm
I20260928 15:00:54.561599 16466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:00:54 dist-test-slave-j968 krb5kdc[15015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607654, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:00:54.564734 16466 init.cc:377] Logged in from keytab as kudu/127.31.40.3@KRBTEST.COM (short username kudu)
I20260928 15:00:54.566337 16466 webserver.cc:460] Webserver started at http://127.31.40.3:33821/ using document root <none> and password file <none>
I20260928 15:00:54.567087 16466 fs_manager.cc:359] Metadata directory not provided
I20260928 15:00:54.567220 16466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:00:54.572765 16466 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 15:00:54.575688 16481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:54.577167 16466 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 15:00:54.577353 16466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-2/wal
uuid: "e3b9cbb8fe0945beb891b825281d5bfb"
format_stamp: "Formatted at 2026-09-28 15:00:08 on dist-test-slave-j968"
server_key: "3190af6063ecb0bbbd71d9775df3a410"
server_key_iv: "1e56583a12a5b4b0eb45352f18a991b5"
server_key_version: "encryptionkey@0"
I20260928 15:00:54.578434 16466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:00:54.594023 16466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:00:54.598034 16466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:00:54.598302 16466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:00:54.599262 16466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:00:54.600652 16466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:00:54.600767 16466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:54.600895 16466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:00:54.600980 16466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:54.634313 16466 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.3:46849
I20260928 15:00:54.634371 16593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.3:46849 every 8 connection(s)
I20260928 15:00:54.635787 16466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
I20260928 15:00:54.637876 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 16466
I20260928 15:00:54.641464 16597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:54.639308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34035 (local address 127.31.40.3:41505)
0928 15:00:54.640280 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:54.640533 (+ 253us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":702,"thread_start_us":375,"threads_started":1}
W20260928 15:00:54.642138 16594 heartbeater.cc:644] Failed to heartbeat to 127.31.40.62:34035 (0 consecutive failures): Network error: Failed to ping master at 127.31.40.62:34035: Client connection negotiation failed: client connection to 127.31.40.62:34035: connect: Connection refused (error 111)
I20260928 15:00:54.642915 16597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:54.642612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34035 (local address 127.31.40.3:54551)
0928 15:00:54.642738 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:54.642803 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 15:00:54.643715 16597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:54.643406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34035 (local address 127.31.40.3:37933)
0928 15:00:54.643539 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:54.643618 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 15:00:54.644018 16594 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 15:00:54.733116 16598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:54.732330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34035 (local address 127.31.40.1:54557)
0928 15:00:54.732797 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:54.732887 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":221,"threads_started":1}
I20260928 15:00:55.186093 16599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:55.185296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34035 (local address 127.31.40.2:36341)
0928 15:00:55.185740 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:55.185845 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":209,"threads_started":1}
I20260928 15:00:55.645304 16600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:55.644487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34035 (local address 127.31.40.3:56633)
0928 15:00:55.644962 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:55.645070 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":209,"threads_started":1}
I20260928 15:00:55.735062 16601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:55.734027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34035 (local address 127.31.40.1:54825)
0928 15:00:55.734653 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:55.734800 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":257,"threads_started":1}
I20260928 15:00:56.187906 16602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:56.187102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34035 (local address 127.31.40.2:41365)
0928 15:00:56.187613 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:56.187699 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":246,"threads_started":1}
I20260928 15:00:56.647006 16603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:56.646278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34035 (local address 127.31.40.3:39793)
0928 15:00:56.646751 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:56.646838 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":227,"threads_started":1}
I20260928 15:00:56.736990 16604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:56.736177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34035 (local address 127.31.40.1:57399)
0928 15:00:56.736724 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:56.736809 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":302,"threads_started":1}
I20260928 15:00:57.189603 16605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:57.188837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34035 (local address 127.31.40.2:49705)
0928 15:00:57.189323 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:57.189408 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":242,"threads_started":1}
I20260928 15:00:57.648569 16606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:57.647854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34035 (local address 127.31.40.3:43301)
0928 15:00:57.648324 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:57.648402 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":221,"threads_started":1}
I20260928 15:00:57.738659 16607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:57.737937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34035 (local address 127.31.40.1:41745)
0928 15:00:57.738399 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:57.738491 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":217,"threads_started":1}
I20260928 15:00:58.191339 16608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:58.190553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34035 (local address 127.31.40.2:41271)
0928 15:00:58.191037 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:58.191121 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":240,"threads_started":1}
I20260928 15:00:58.650233 16609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:58.649405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34035 (local address 127.31.40.3:42647)
0928 15:00:58.649955 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:58.650039 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":221,"threads_started":1}
I20260928 15:00:58.740567 16610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:58.739561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34035 (local address 127.31.40.1:58545)
0928 15:00:58.740254 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:58.740382 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":397,"threads_started":1}
I20260928 15:00:59.193264 16611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:59.192409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34035 (local address 127.31.40.2:43195)
0928 15:00:59.192980 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:59.193075 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":347,"threads_started":1}
I20260928 15:00:59.652029 16612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:59.651178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34035 (local address 127.31.40.3:53483)
0928 15:00:59.651712 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:59.651795 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":300,"threads_started":1}
I20260928 15:00:59.742306 16613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:59.741568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34035 (local address 127.31.40.1:59149)
0928 15:00:59.742043 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:59.742134 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":259,"threads_started":1}
I20260928 15:01:00.195122 16614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:00.194337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34035 (local address 127.31.40.2:56049)
0928 15:01:00.194852 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:00.194948 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":270,"threads_started":1}
I20260928 15:01:00.653584 16615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:00.652884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34035 (local address 127.31.40.3:50373)
0928 15:01:00.653331 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:00.653421 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":233,"threads_started":1}
I20260928 15:01:00.744563 16616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:00.743589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34035 (local address 127.31.40.1:44923)
0928 15:01:00.744245 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:00.744343 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":370,"threads_started":1}
I20260928 15:01:01.196965 16617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:01.196152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34035 (local address 127.31.40.2:55633)
0928 15:01:01.196665 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:01.196746 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":244,"threads_started":1}
I20260928 15:01:01.655519 16618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:01.654599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34035 (local address 127.31.40.3:53811)
0928 15:01:01.655203 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:01.655301 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":328,"threads_started":1}
I20260928 15:01:01.746335 16619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:01.745586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34035 (local address 127.31.40.1:55905)
0928 15:01:01.746065 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:01.746146 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":269,"threads_started":1}
I20260928 15:01:02.198874 16620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:02.198062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34035 (local address 127.31.40.2:44823)
0928 15:01:02.198560 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:02.198647 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":250,"threads_started":1}
I20260928 15:01:02.657208 16621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:02.656473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34035 (local address 127.31.40.3:40919)
0928 15:01:02.656952 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:02.657038 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":217,"threads_started":1}
I20260928 15:01:02.748298 16622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:02.747447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34035 (local address 127.31.40.1:46157)
0928 15:01:02.748004 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:02.748098 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":241,"threads_started":1}
I20260928 15:01:03.200595 16623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:03.199786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34035 (local address 127.31.40.2:37843)
0928 15:01:03.200326 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:03.200418 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":254,"threads_started":1}
I20260928 15:01:03.659138 16624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:03.658317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34035 (local address 127.31.40.3:38085)
0928 15:01:03.658852 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:03.658947 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":258,"threads_started":1}
I20260928 15:01:03.750324 16625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:03.749533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34035 (local address 127.31.40.1:56021)
0928 15:01:03.750046 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:03.750136 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":278,"threads_started":1}
I20260928 15:01:04.202477 16626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:04.201606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34035 (local address 127.31.40.2:52377)
0928 15:01:04.202163 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:04.202283 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":278,"threads_started":1}
I20260928 15:01:04.660935 16627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:04.660179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34035 (local address 127.31.40.3:35695)
0928 15:01:04.660646 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:04.660731 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":232,"threads_started":1}
I20260928 15:01:04.752236 16628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:04.751399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34035 (local address 127.31.40.1:57573)
0928 15:01:04.751951 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:04.752039 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":230,"threads_started":1}
I20260928 15:01:05.204408 16629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:05.203528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34035 (local address 127.31.40.2:58679)
0928 15:01:05.204091 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:05.204189 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":271,"threads_started":1}
I20260928 15:01:05.662612 16630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:05.661886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34035 (local address 127.31.40.3:55963)
0928 15:01:05.662334 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:05.662429 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":216,"threads_started":1}
I20260928 15:01:05.754384 16631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:05.753398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34035 (local address 127.31.40.1:56151)
0928 15:01:05.754064 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:05.754150 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":369,"threads_started":1}
I20260928 15:01:06.206228 16632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:06.205448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34035 (local address 127.31.40.2:37605)
0928 15:01:06.205979 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:06.206065 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":234,"threads_started":1}
I20260928 15:01:06.664439 16633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:06.663678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34035 (local address 127.31.40.3:38993)
0928 15:01:06.664175 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:06.664254 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":269,"threads_started":1}
I20260928 15:01:06.756415 16634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:06.755526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34035 (local address 127.31.40.1:33797)
0928 15:01:06.756094 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:06.756192 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":271,"threads_started":1}
I20260928 15:01:07.208076 16635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:07.207202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34035 (local address 127.31.40.2:52537)
0928 15:01:07.207721 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:07.207811 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":284,"threads_started":1}
I20260928 15:01:07.666230 16636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:07.665425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34035 (local address 127.31.40.3:35895)
0928 15:01:07.665933 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:07.666038 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":238,"threads_started":1}
I20260928 15:01:07.758142 16637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:07.757407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34035 (local address 127.31.40.1:33879)
0928 15:01:07.757873 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:07.757956 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":228,"threads_started":1}
I20260928 15:01:08.209758 16638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:08.209038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34035 (local address 127.31.40.2:47511)
0928 15:01:08.209480 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:08.209570 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":243,"threads_started":1}
I20260928 15:01:08.668087 16639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:08.667166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34035 (local address 127.31.40.3:49799)
0928 15:01:08.667678 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:08.667775 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":269,"threads_started":1}
I20260928 15:01:08.760190 16640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:08.759296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34035 (local address 127.31.40.1:41133)
0928 15:01:08.759755 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:08.759845 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":248,"threads_started":1}
I20260928 15:01:09.211853 16641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:09.211020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34035 (local address 127.31.40.2:52631)
0928 15:01:09.211521 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:09.211623 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":252,"threads_started":1}
I20260928 15:01:09.645061 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 16164
I20260928 15:01:09.665093 16642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:09.646309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55518 (local address 127.31.40.1:32837)
0928 15:01:09.647082 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:09.647097 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:09.647137 (+ 40us) server_negotiation.cc:407] Connection header received
0928 15:01:09.648951 (+ 1814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:09.648962 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:09.649446 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:09.649772 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:09.651485 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:09.652949 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:09.654329 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:09.655080 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:09.655240 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:09.658345 (+ 3105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:09.658355 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:09.658374 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:09.658417 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:09.661744 (+ 3327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:09.662488 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:09.662495 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:09.662502 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:09.662676 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:09.663024 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:09.663027 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:09.663030 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:09.663595 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:09.663844 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:09.664331 (+ 487us) server_negotiation.cc:299] Negotiation successful
0928 15:01:09.664790 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":438,"threads_started":1}
I20260928 15:01:09.666700 16327 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:01:09.672745 16644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:09.669177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34035 (local address 127.31.40.3:36727)
0928 15:01:09.672417 (+ 3240us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:09.672561 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.62:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3035,"thread_start_us":2891,"threads_started":1}
I20260928 15:01:09.754868 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 16164
I20260928 15:01:09.772658 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 16335
I20260928 15:01:09.791148 16645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:09.773667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50230 (local address 127.31.40.2:32855)
0928 15:01:09.774185 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:09.774200 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:09.774239 (+ 39us) server_negotiation.cc:407] Connection header received
0928 15:01:09.776013 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:09.776024 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:09.776488 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:09.776787 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:09.778241 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:09.779515 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:09.780991 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:09.781766 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:09.781949 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:09.784773 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:09.784784 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:09.784808 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:09.784870 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:09.788032 (+ 3162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:09.788683 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:09.788690 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:09.788697 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:09.788856 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:09.789229 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:09.789232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:09.789234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:09.789794 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:09.790032 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:09.790423 (+ 391us) server_negotiation.cc:299] Negotiation successful
0928 15:01:09.790864 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":246,"threads_started":1}
I20260928 15:01:09.792742 16458 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:01:09.877780 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 16335
I20260928 15:01:09.896368 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 16466
I20260928 15:01:09.915108 16647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:09.897530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37034 (local address 127.31.40.3:46849)
0928 15:01:09.898238 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:09.898253 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:09.898301 (+ 48us) server_negotiation.cc:407] Connection header received
0928 15:01:09.900072 (+ 1771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:09.900082 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:09.900575 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:09.900887 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:09.902253 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:09.903582 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:09.904947 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:09.905669 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:09.905857 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:09.908505 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:09.908512 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:09.908528 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:09.908578 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:09.911615 (+ 3037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:09.912339 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:09.912364 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:09.912372 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:09.912544 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:09.912894 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:09.912897 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:09.912900 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:09.913475 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:09.913708 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:09.914249 (+ 541us) server_negotiation.cc:299] Negotiation successful
0928 15:01:09.914708 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":396,"threads_started":1}
I20260928 15:01:09.916676 16589 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:01:10.001711 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 16466
2026-09-28T15:01:10Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64891 ms)
[----------] 30 tests from SecurityITest (304602 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:01:10 dist-test-slave-j968 krb5kdc[16651](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:01:10 dist-test-slave-j968 krb5kdc[16651](info): set up 2 sockets
Sep 28 15:01:10 dist-test-slave-j968 krb5kdc[16651](info): commencing operation
krb5kdc: starting...
W20260928 15:01:11.125044 31904 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Sep 28 15:01:11 dist-test-slave-j968 krb5kdc[16651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:01:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:01:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab.
Entry for principal kudu/127.31.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab.
I20260928 15:01:11.384092 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:38363
--webserver_interface=127.31.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:46127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.40.62:38363
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:01:11.721480 16664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:11.721896 16664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:11.722067 16664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:11.729441 16664 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:01:11.729598 16664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:11.729704 16664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:11.729816 16664 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:01:11.729904 16664 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:01:11.740083 16664 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:46127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.40.62:38363
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:38363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:01:11.741521 16664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:11.742944 16664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:11.755481 16669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:11.755582 16670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:11.755509 16672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:11.755858 16664 server_base.cc:1034] running on GCE node
I20260928 15:01:11.757030 16664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:11.758461 16664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:11.759670 16664 hybrid_clock.cc:648] HybridClock initialized: now 1790607671759595 us; error 62 us; skew 500 ppm
I20260928 15:01:11.760067 16664 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:01:11 dist-test-slave-j968 krb5kdc[16651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607671, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:01:11.771049 16664 init.cc:377] Logged in from keytab as kudu/127.31.40.62@KRBTEST.COM (short username kudu)
I20260928 15:01:11.772904 16664 webserver.cc:460] Webserver started at http://127.31.40.62:40361/ using document root <none> and password file <none>
I20260928 15:01:11.773666 16664 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:11.773789 16664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:11.774060 16664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:11.777328 16664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/master-0/data/instance:
uuid: "54ed0dec12464877825c48ee85b5b65c"
format_stamp: "Formatted at 2026-09-28 15:01:11 on dist-test-slave-j968"
server_key: "d77251bd6826d00a2c7c3051e400ae16"
server_key_iv: "bbb8bea5d503e1a7e2078e07d1d4f9c6"
server_key_version: "encryptionkey@0"
I20260928 15:01:11.778066 16664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/master-0/wal/instance:
uuid: "54ed0dec12464877825c48ee85b5b65c"
format_stamp: "Formatted at 2026-09-28 15:01:11 on dist-test-slave-j968"
server_key: "d77251bd6826d00a2c7c3051e400ae16"
server_key_iv: "bbb8bea5d503e1a7e2078e07d1d4f9c6"
server_key_version: "encryptionkey@0"
I20260928 15:01:11.783555 16664 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260928 15:01:11.787314 16679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:11.788916 16664 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:01:11.789121 16664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/master-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/master-0/wal
uuid: "54ed0dec12464877825c48ee85b5b65c"
format_stamp: "Formatted at 2026-09-28 15:01:11 on dist-test-slave-j968"
server_key: "d77251bd6826d00a2c7c3051e400ae16"
server_key_iv: "bbb8bea5d503e1a7e2078e07d1d4f9c6"
server_key_version: "encryptionkey@0"
I20260928 15:01:11.789336 16664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:11.805197 16664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:11.809437 16664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:11.809739 16664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:11.828881 16664 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.62:38363
I20260928 15:01:11.828919 16730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.62:38363 every 8 connection(s)
I20260928 15:01:11.830376 16664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
I20260928 15:01:11.834270 16731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:11.838342 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 16664
I20260928 15:01:11.838577 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/master-0/wal/instance
I20260928 15:01:11.839252 31904 external_mini_cluster.cc:1428] Setting key fd587b97420cfa2006561a7bce2a843c
I20260928 15:01:11.848340 16731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54ed0dec12464877825c48ee85b5b65c: Bootstrap starting.
I20260928 15:01:11.852248 16731 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54ed0dec12464877825c48ee85b5b65c: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:11.853624 16731 log.cc:824] T 00000000000000000000000000000000 P 54ed0dec12464877825c48ee85b5b65c: Log is configured to *not* fsync() on all Append() calls
Sep 28 15:01:11 dist-test-slave-j968 krb5kdc[16651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 15:01:11.856724 16731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54ed0dec12464877825c48ee85b5b65c: No bootstrap required, opened a new log
I20260928 15:01:11.863737 16734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:11.841231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55858 (local address 127.31.40.62:38363)
0928 15:01:11.842127 (+ 896us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:11.842139 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:11.842182 (+ 43us) server_negotiation.cc:407] Connection header received
0928 15:01:11.844026 (+ 1844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:11.844051 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:11.844481 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:11.844817 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:11.846632 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:11.848034 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:11.849977 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:11.850679 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:11.850809 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:11.855829 (+ 5020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:11.855837 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:11.855852 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:11.855965 (+ 113us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:11.859194 (+ 3229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:11.860147 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:11.860154 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:11.860164 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:11.860341 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:11.860760 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:11.860767 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:11.860775 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:11.861363 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:11.861739 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:11.862194 (+ 455us) server_negotiation.cc:299] Negotiation successful
0928 15:01:11.862659 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":338,"threads_started":1}
I20260928 15:01:11.864765 16731 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54ed0dec12464877825c48ee85b5b65c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54ed0dec12464877825c48ee85b5b65c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 38363 } }
I20260928 15:01:11.865177 16731 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54ed0dec12464877825c48ee85b5b65c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:11.865326 16731 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54ed0dec12464877825c48ee85b5b65c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54ed0dec12464877825c48ee85b5b65c, State: Initialized, Role: FOLLOWER
I20260928 15:01:11.865949 16731 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54ed0dec12464877825c48ee85b5b65c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54ed0dec12464877825c48ee85b5b65c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 38363 } }
I20260928 15:01:11.866214 16731 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54ed0dec12464877825c48ee85b5b65c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:01:11.866364 16731 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54ed0dec12464877825c48ee85b5b65c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:01:11.866568 16731 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54ed0dec12464877825c48ee85b5b65c [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:11.868790 16731 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54ed0dec12464877825c48ee85b5b65c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54ed0dec12464877825c48ee85b5b65c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 38363 } }
I20260928 15:01:11.869234 16731 leader_election.cc:302] T 00000000000000000000000000000000 P 54ed0dec12464877825c48ee85b5b65c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54ed0dec12464877825c48ee85b5b65c; no voters:
I20260928 15:01:11.869752 16731 leader_election.cc:288] T 00000000000000000000000000000000 P 54ed0dec12464877825c48ee85b5b65c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:01:11.869906 16736 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54ed0dec12464877825c48ee85b5b65c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:01:11.870625 16736 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54ed0dec12464877825c48ee85b5b65c [term 1 LEADER]: Becoming Leader. State: Replica: 54ed0dec12464877825c48ee85b5b65c, State: Running, Role: LEADER
I20260928 15:01:11.871153 16736 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54ed0dec12464877825c48ee85b5b65c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54ed0dec12464877825c48ee85b5b65c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 38363 } }
I20260928 15:01:11.872053 16731 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54ed0dec12464877825c48ee85b5b65c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:01:11.873924 16738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54ed0dec12464877825c48ee85b5b65c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54ed0dec12464877825c48ee85b5b65c. Latest consensus state: current_term: 1 leader_uuid: "54ed0dec12464877825c48ee85b5b65c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54ed0dec12464877825c48ee85b5b65c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 38363 } } }
I20260928 15:01:11.874254 16738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54ed0dec12464877825c48ee85b5b65c [sys.catalog]: This master's current role is: LEADER
I20260928 15:01:11.875094 16737 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54ed0dec12464877825c48ee85b5b65c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54ed0dec12464877825c48ee85b5b65c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54ed0dec12464877825c48ee85b5b65c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 38363 } } }
I20260928 15:01:11.875435 16737 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54ed0dec12464877825c48ee85b5b65c [sys.catalog]: This master's current role is: LEADER
I20260928 15:01:11.877765 16741 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:01:11.883164 16741 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:01:11.892633 16741 catalog_manager.cc:1347] Generated new cluster ID: 70d9a80a16b0464987b9912c1c73c1b0
I20260928 15:01:11.892792 16741 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:01:11.918133 16741 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:01:11.919842 16741 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:01:11.930186 16741 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54ed0dec12464877825c48ee85b5b65c: Generated new TSK 0
I20260928 15:01:11.931305 16741 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.40.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab.
Entry for principal kudu/127.31.40.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab.
I20260928 15:01:12.000722 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.1:0
--local_ip_for_outbound_sockets=127.31.40.1
--webserver_interface=127.31.40.1
--webserver_port=0
--tserver_master_addrs=127.31.40.62:38363
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab
--principal=kudu/127.31.40.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:46127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:01:12.351552 16757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:12.351850 16757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:12.352093 16757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:12.359762 16757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:12.359942 16757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:12.360145 16757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.1
I20260928 15:01:12.370386 16757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:46127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab
--principal=kudu/127.31.40.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.40.1
--webserver_port=0
--tserver_master_addrs=127.31.40.62:38363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.1
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:01:12.371757 16757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:12.373135 16757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:12.386754 16765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:12.386660 16762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:12.386686 16763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:12.387514 16757 server_base.cc:1034] running on GCE node
I20260928 15:01:12.388171 16757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:12.389106 16757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:12.390338 16757 hybrid_clock.cc:648] HybridClock initialized: now 1790607672390276 us; error 51 us; skew 500 ppm
I20260928 15:01:12.390648 16757 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:01:12 dist-test-slave-j968 krb5kdc[16651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607672, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:01:12.393903 16757 init.cc:377] Logged in from keytab as kudu/127.31.40.1@KRBTEST.COM (short username kudu)
I20260928 15:01:12.395421 16757 webserver.cc:460] Webserver started at http://127.31.40.1:35441/ using document root <none> and password file <none>
I20260928 15:01:12.396145 16757 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:12.396247 16757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:12.396533 16757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:12.399474 16757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/ts-0/data/instance:
uuid: "ad269aacd7db4441a4ffd3fd1502cabb"
format_stamp: "Formatted at 2026-09-28 15:01:12 on dist-test-slave-j968"
server_key: "0992afe1918c470721fcf5966a284569"
server_key_iv: "86362a0b3e29e9020077abb79806a125"
server_key_version: "encryptionkey@0"
I20260928 15:01:12.400306 16757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/ts-0/wal/instance:
uuid: "ad269aacd7db4441a4ffd3fd1502cabb"
format_stamp: "Formatted at 2026-09-28 15:01:12 on dist-test-slave-j968"
server_key: "0992afe1918c470721fcf5966a284569"
server_key_iv: "86362a0b3e29e9020077abb79806a125"
server_key_version: "encryptionkey@0"
I20260928 15:01:12.405767 16757 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260928 15:01:12.409382 16772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:12.410787 16757 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260928 15:01:12.410993 16757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/ts-0/wal
uuid: "ad269aacd7db4441a4ffd3fd1502cabb"
format_stamp: "Formatted at 2026-09-28 15:01:12 on dist-test-slave-j968"
server_key: "0992afe1918c470721fcf5966a284569"
server_key_iv: "86362a0b3e29e9020077abb79806a125"
server_key_version: "encryptionkey@0"
I20260928 15:01:12.411204 16757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:12.430163 16757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:12.434685 16757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:12.434996 16757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:12.436246 16757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:01:12.437634 16757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:01:12.437707 16757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:12.437860 16757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:01:12.437932 16757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:12.471387 16757 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.1:35705
I20260928 15:01:12.471434 16884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.1:35705 every 8 connection(s)
I20260928 15:01:12.472913 16757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
I20260928 15:01:12.476773 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 16757
I20260928 15:01:12.477036 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/ts-0/wal/instance
I20260928 15:01:12.477875 31904 external_mini_cluster.cc:1428] Setting key 23b885cbbba66d2d0bd6dfbc40026f43
Sep 28 15:01:12 dist-test-slave-j968 krb5kdc[16651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607672, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.1@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 15:01:12.495910 16887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:12.475739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:47733 (local address 127.31.40.62:38363)
0928 15:01:12.476241 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:12.476245 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:12.477716 (+ 1471us) server_negotiation.cc:407] Connection header received
0928 15:01:12.480136 (+ 2420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:12.480142 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:12.480238 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:12.480378 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:12.482108 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:12.482936 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:12.484804 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:12.485260 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:12.485415 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:12.489508 (+ 4093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:12.489514 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:12.489517 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:12.489552 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:12.492515 (+ 2963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:12.493315 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:12.493324 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:12.493328 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:12.493463 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:12.493943 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:12.493948 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:12.493954 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:12.494249 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:12.494432 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:12.495204 (+ 772us) server_negotiation.cc:299] Negotiation successful
0928 15:01:12.495467 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":280,"threads_started":1}
I20260928 15:01:12.497010 16888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:12.476488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:38363 (local address 127.31.40.1:47733)
0928 15:01:12.477471 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:12.477584 (+ 113us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:12.479795 (+ 2211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:12.480653 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:12.480663 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:12.481245 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:01:12.481918 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:12.481940 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:12.483141 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:12.483146 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:12.484637 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:12.484650 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:12.485533 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:12.485541 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:12.485822 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:12.486622 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:01:12.486654 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:01:12.489261 (+ 2607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:01:12.492670 (+ 3409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:01:12.492680 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:01:12.492699 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:01:12.493161 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:01:12.493585 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:01:12.493592 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:01:12.493596 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:01:12.493811 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:01:12.494402 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:01:12.494410 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:01:12.494784 (+ 374us) client_negotiation.cc:769] Sending connection context
0928 15:01:12.495081 (+ 297us) client_negotiation.cc:240] Negotiation successful
0928 15:01:12.495677 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":410,"threads_started":1}
I20260928 15:01:12.498848 16885 heartbeater.cc:347] Connected to a master server at 127.31.40.62:38363
I20260928 15:01:12.499225 16885 heartbeater.cc:464] Registering TS with master...
I20260928 15:01:12.500218 16885 heartbeater.cc:511] Master 127.31.40.62:38363 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.31.40.2@KRBTEST.COM; defaulting to no policy
I20260928 15:01:12.502825 16696 ts_manager.cc:194] Registered new tserver with Master: ad269aacd7db4441a4ffd3fd1502cabb (127.31.40.1:35705)
I20260928 15:01:12.504732 16696 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.1@KRBTEST.COM'} at 127.31.40.1:47733
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab.
Entry for principal kudu/127.31.40.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab.
I20260928 15:01:12.535976 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.2:0
--local_ip_for_outbound_sockets=127.31.40.2
--webserver_interface=127.31.40.2
--webserver_port=0
--tserver_master_addrs=127.31.40.62:38363
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab
--principal=kudu/127.31.40.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:46127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:01:12.873449 16891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:12.873800 16891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:12.873905 16891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:12.881608 16891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:12.881754 16891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:12.881879 16891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.2
I20260928 15:01:12.892002 16891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:46127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab
--principal=kudu/127.31.40.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.40.2
--webserver_port=0
--tserver_master_addrs=127.31.40.62:38363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.2
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:01:12.893285 16891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:12.894544 16891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:12.908052 16896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:12.908043 16897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:12.908068 16899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:12.909265 16891 server_base.cc:1034] running on GCE node
I20260928 15:01:12.909919 16891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:12.910916 16891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:12.912143 16891 hybrid_clock.cc:648] HybridClock initialized: now 1790607672912084 us; error 53 us; skew 500 ppm
I20260928 15:01:12.912482 16891 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:01:12 dist-test-slave-j968 krb5kdc[16651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607672, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:01:12.915856 16891 init.cc:377] Logged in from keytab as kudu/127.31.40.2@KRBTEST.COM (short username kudu)
I20260928 15:01:12.917538 16891 webserver.cc:460] Webserver started at http://127.31.40.2:35033/ using document root <none> and password file <none>
I20260928 15:01:12.918241 16891 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:12.918373 16891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:12.918704 16891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:12.921677 16891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/ts-1/data/instance:
uuid: "ec57a9abba334534901d01f3ecf1c715"
format_stamp: "Formatted at 2026-09-28 15:01:12 on dist-test-slave-j968"
server_key: "bee3d9a3225cfc4fc7e3c6ff311bd5c9"
server_key_iv: "763f5a199d660d5d32b4165464cd66c9"
server_key_version: "encryptionkey@0"
I20260928 15:01:12.922502 16891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/ts-1/wal/instance:
uuid: "ec57a9abba334534901d01f3ecf1c715"
format_stamp: "Formatted at 2026-09-28 15:01:12 on dist-test-slave-j968"
server_key: "bee3d9a3225cfc4fc7e3c6ff311bd5c9"
server_key_iv: "763f5a199d660d5d32b4165464cd66c9"
server_key_version: "encryptionkey@0"
I20260928 15:01:12.927963 16891 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 15:01:12.931432 16906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:12.932830 16891 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 15:01:12.933008 16891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/ts-1/wal
uuid: "ec57a9abba334534901d01f3ecf1c715"
format_stamp: "Formatted at 2026-09-28 15:01:12 on dist-test-slave-j968"
server_key: "bee3d9a3225cfc4fc7e3c6ff311bd5c9"
server_key_iv: "763f5a199d660d5d32b4165464cd66c9"
server_key_version: "encryptionkey@0"
I20260928 15:01:12.933205 16891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:12.944079 16891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:12.948333 16891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:12.948628 16891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:12.949582 16891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:01:12.950855 16891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:01:12.950934 16891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:12.951062 16891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:01:12.951112 16891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:12.985062 16891 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.2:44189
I20260928 15:01:12.985110 17018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.2:44189 every 8 connection(s)
I20260928 15:01:12.986403 16891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
I20260928 15:01:12.992432 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 16891
I20260928 15:01:12.992631 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/ts-1/wal/instance
I20260928 15:01:12.993112 31904 external_mini_cluster.cc:1428] Setting key 94c9f3890876d665edc9ecd51b31ffe3
Sep 28 15:01:12 dist-test-slave-j968 krb5kdc[16651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607672, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.2@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 15:01:13.008584 16887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:12.989115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:47729 (local address 127.31.40.62:38363)
0928 15:01:12.989306 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:12.989311 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:12.990491 (+ 1180us) server_negotiation.cc:407] Connection header received
0928 15:01:12.992383 (+ 1892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:12.992389 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:12.992485 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:12.992608 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:12.994561 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:12.995413 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:12.997044 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:12.997477 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:12.997614 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:13.001538 (+ 3924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:13.001543 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:13.001547 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:13.001581 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:13.005101 (+ 3520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:13.005966 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:13.005972 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:13.005975 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:13.006111 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:13.006607 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:13.006617 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:13.006620 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:13.006915 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:13.007118 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:13.007935 (+ 817us) server_negotiation.cc:299] Negotiation successful
0928 15:01:13.008197 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 15:01:13.009687 17021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:12.989604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:38363 (local address 127.31.40.2:47729)
0928 15:01:12.990311 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:12.990349 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:12.992121 (+ 1772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:12.993047 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:12.993055 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:12.993649 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:01:12.994391 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:12.994410 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:12.995564 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:12.995569 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:12.996885 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:12.996897 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:12.997702 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:12.997709 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:12.997939 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:12.998693 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:01:12.998792 (+ 99us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:01:13.001274 (+ 2482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:01:13.005277 (+ 4003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:01:13.005288 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:01:13.005308 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:01:13.005811 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:01:13.006248 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:01:13.006254 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:01:13.006258 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:01:13.006472 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:01:13.007082 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:01:13.007091 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:01:13.007486 (+ 395us) client_negotiation.cc:769] Sending connection context
0928 15:01:13.007781 (+ 295us) client_negotiation.cc:240] Negotiation successful
0928 15:01:13.008422 (+ 641us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":281,"threads_started":1}
I20260928 15:01:13.011435 17019 heartbeater.cc:347] Connected to a master server at 127.31.40.62:38363
I20260928 15:01:13.011785 17019 heartbeater.cc:464] Registering TS with master...
I20260928 15:01:13.012598 17019 heartbeater.cc:511] Master 127.31.40.62:38363 requested a full tablet report, sending...
I20260928 15:01:13.014178 16696 ts_manager.cc:194] Registered new tserver with Master: ec57a9abba334534901d01f3ecf1c715 (127.31.40.2:44189)
I20260928 15:01:13.015103 16696 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.2@KRBTEST.COM'} at 127.31.40.2:47729
WARNING: no policy specified for kudu/127.31.40.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab.
Entry for principal kudu/127.31.40.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab.
I20260928 15:01:13.051939 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.3:0
--local_ip_for_outbound_sockets=127.31.40.3
--webserver_interface=127.31.40.3
--webserver_port=0
--tserver_master_addrs=127.31.40.62:38363
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab
--principal=kudu/127.31.40.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:46127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:01:13.387441 17024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:13.387720 17024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:13.387946 17024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:13.395403 17024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:13.395507 17024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:13.395617 17024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.3
I20260928 15:01:13.405761 17024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:46127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab
--principal=kudu/127.31.40.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.40.3
--webserver_port=0
--tserver_master_addrs=127.31.40.62:38363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.3
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:01:13.407068 17024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:13.408463 17024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:13.421775 17030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:13.421729 17029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:13.421810 17032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:13.422585 17024 server_base.cc:1034] running on GCE node
I20260928 15:01:13.423251 17024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:13.424356 17024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:13.425612 17024 hybrid_clock.cc:648] HybridClock initialized: now 1790607673425563 us; error 45 us; skew 500 ppm
I20260928 15:01:13.425967 17024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:01:13 dist-test-slave-j968 krb5kdc[16651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607673, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:01:13.429497 17024 init.cc:377] Logged in from keytab as kudu/127.31.40.3@KRBTEST.COM (short username kudu)
I20260928 15:01:13.431028 17024 webserver.cc:460] Webserver started at http://127.31.40.3:34689/ using document root <none> and password file <none>
I20260928 15:01:13.431727 17024 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:13.431829 17024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:13.432175 17024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:13.435174 17024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/ts-2/data/instance:
uuid: "10bae2153c1c4d30b26aae53cff4d846"
format_stamp: "Formatted at 2026-09-28 15:01:13 on dist-test-slave-j968"
server_key: "e36166a63c23dca9f647a2985bd20e09"
server_key_iv: "7b405b81a5b924aa66b1c457f2c9367d"
server_key_version: "encryptionkey@0"
I20260928 15:01:13.436007 17024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/ts-2/wal/instance:
uuid: "10bae2153c1c4d30b26aae53cff4d846"
format_stamp: "Formatted at 2026-09-28 15:01:13 on dist-test-slave-j968"
server_key: "e36166a63c23dca9f647a2985bd20e09"
server_key_iv: "7b405b81a5b924aa66b1c457f2c9367d"
server_key_version: "encryptionkey@0"
I20260928 15:01:13.441617 17024 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 15:01:13.445189 17039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:13.446579 17024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260928 15:01:13.446807 17024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/ts-2/wal
uuid: "10bae2153c1c4d30b26aae53cff4d846"
format_stamp: "Formatted at 2026-09-28 15:01:13 on dist-test-slave-j968"
server_key: "e36166a63c23dca9f647a2985bd20e09"
server_key_iv: "7b405b81a5b924aa66b1c457f2c9367d"
server_key_version: "encryptionkey@0"
I20260928 15:01:13.447021 17024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:13.459450 17024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:13.463755 17024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:13.464079 17024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:13.465068 17024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:01:13.466480 17024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:01:13.466552 17024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:13.466696 17024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:01:13.466755 17024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:13.502151 17024 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.3:40647
I20260928 15:01:13.502241 17151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.3:40647 every 8 connection(s)
I20260928 15:01:13.503484 17024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
I20260928 15:01:13.506979 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 17024
I20260928 15:01:13.507159 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/ts-2/wal/instance
I20260928 15:01:13.507649 31904 external_mini_cluster.cc:1428] Setting key c94b4c8c1609f683dc6d88b271f82423
I20260928 15:01:13.508818 16885 heartbeater.cc:503] Master 127.31.40.62:38363 was elected leader, sending a full tablet report...
Sep 28 15:01:13 dist-test-slave-j968 krb5kdc[16651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607673, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.3@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 15:01:13.524394 16887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:13.506362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:60729 (local address 127.31.40.62:38363)
0928 15:01:13.506604 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:13.506610 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:13.508057 (+ 1447us) server_negotiation.cc:407] Connection header received
0928 15:01:13.510473 (+ 2416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:13.510481 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:13.510563 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:13.510638 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:13.512358 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:13.513113 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:13.514776 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:13.515179 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:13.515283 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:13.518589 (+ 3306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:13.518594 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:13.518597 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:13.518627 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:13.521284 (+ 2657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:13.522008 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:13.522013 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:13.522016 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:13.522200 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:13.522723 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:13.522727 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:13.522729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:13.523057 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:13.523228 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:13.523982 (+ 754us) server_negotiation.cc:299] Negotiation successful
0928 15:01:13.524139 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 15:01:13.525352 17154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:13.506941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:38363 (local address 127.31.40.3:60729)
0928 15:01:13.507853 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:13.507919 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:13.510131 (+ 2212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:13.511034 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:13.511041 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:13.511560 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:01:13.512211 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:13.512229 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:13.513253 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:13.513257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:13.514596 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:13.514608 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:13.515412 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:13.515422 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:13.515625 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:13.516300 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:01:13.516322 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:01:13.518413 (+ 2091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:01:13.521426 (+ 3013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:01:13.521435 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:01:13.521452 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:01:13.521905 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:01:13.522326 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:01:13.522330 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:01:13.522333 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:01:13.522593 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:01:13.523195 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:01:13.523205 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:01:13.523593 (+ 388us) client_negotiation.cc:769] Sending connection context
0928 15:01:13.523805 (+ 212us) client_negotiation.cc:240] Negotiation successful
0928 15:01:13.524237 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":408,"threads_started":1}
I20260928 15:01:13.526597 17152 heartbeater.cc:347] Connected to a master server at 127.31.40.62:38363
I20260928 15:01:13.526899 17152 heartbeater.cc:464] Registering TS with master...
I20260928 15:01:13.527660 17152 heartbeater.cc:511] Master 127.31.40.62:38363 requested a full tablet report, sending...
I20260928 15:01:13.529175 16695 ts_manager.cc:194] Registered new tserver with Master: 10bae2153c1c4d30b26aae53cff4d846 (127.31.40.3:40647)
I20260928 15:01:13.530030 16695 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.3@KRBTEST.COM'} at 127.31.40.3:60729
I20260928 15:01:13.535558 31904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:01:13.554430 16887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:13.540164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55868 (local address 127.31.40.62:38363)
0928 15:01:13.540326 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:13.540331 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:13.540871 (+ 540us) server_negotiation.cc:407] Connection header received
0928 15:01:13.540981 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:13.540985 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:13.541055 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:13.541139 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:13.542612 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:13.543348 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:13.544775 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:13.545201 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:13.545297 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:13.547681 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:13.547686 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:13.547689 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:13.547721 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:13.550508 (+ 2787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:13.551578 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:13.551600 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:13.551607 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:13.551780 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:13.552318 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:13.552324 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:13.552327 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:13.552889 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:13.553155 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:13.553729 (+ 574us) server_negotiation.cc:299] Negotiation successful
0928 15:01:13.553972 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 15:01:13.559016 16695 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55868:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:01:13.566169 16695 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:01:13.590615 17164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:13.579112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.2:44189 (local address 127.0.0.1:49204)
0928 15:01:13.579836 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:13.579916 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:13.580164 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:13.581876 (+ 1712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:13.581889 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:13.581950 (+ 61us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:13.582543 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:13.582560 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:13.585056 (+ 2496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:13.585064 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:13.587608 (+ 2544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:13.587623 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:13.589533 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:13.589542 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:13.589792 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:13.589851 (+ 59us) client_negotiation.cc:769] Sending connection context
0928 15:01:13.590003 (+ 152us) client_negotiation.cc:240] Negotiation successful
0928 15:01:13.590193 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"mutex_wait_us":1,"thread_start_us":300,"threads_started":1}
I20260928 15:01:13.591616 17165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:13.579330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49204 (local address 127.31.40.2:44189)
0928 15:01:13.581105 (+ 1775us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:13.581116 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:13.581150 (+ 34us) server_negotiation.cc:407] Connection header received
0928 15:01:13.581255 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:13.581264 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:13.581563 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:13.582809 (+ 1246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:13.583134 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:13.584921 (+ 1787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:13.587803 (+ 2882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:13.589319 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:13.589449 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:13.589664 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:13.590063 (+ 399us) server_negotiation.cc:299] Negotiation successful
0928 15:01:13.590235 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1604,"thread_start_us":1483,"threads_started":1}
I20260928 15:01:13.592509 17168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:13.580379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50866 (local address 127.31.40.1:35705)
0928 15:01:13.584517 (+ 4138us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:13.584527 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:13.584560 (+ 33us) server_negotiation.cc:407] Connection header received
0928 15:01:13.584655 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:13.584663 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:13.584921 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:13.585149 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:13.585828 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:13.587370 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:13.589853 (+ 2483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:13.591284 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:13.591419 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:13.591632 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:13.591940 (+ 308us) server_negotiation.cc:299] Negotiation successful
0928 15:01:13.592113 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3917,"thread_start_us":3699,"threads_started":1}
I20260928 15:01:13.592514 17166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:13.580296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.3:40647 (local address 127.0.0.1:38918)
0928 15:01:13.584704 (+ 4408us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:13.584721 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:13.584845 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:13.585471 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:13.585478 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:13.585510 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:13.586003 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:13.586016 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:13.587997 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:13.588004 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:13.589470 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:13.589484 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:13.591555 (+ 2071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:13.591561 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:13.591706 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:13.591730 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 15:01:13.591799 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 15:01:13.591991 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4297,"mutex_wait_us":2,"thread_start_us":255,"threads_started":1}
I20260928 15:01:13.592542 17167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:13.579762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:35705 (local address 127.0.0.1:50866)
0928 15:01:13.580867 (+ 1105us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:13.580884 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:13.580985 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:13.585141 (+ 4156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:13.585148 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:13.585181 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:13.585634 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:13.585646 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:13.587499 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:13.587506 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:13.589699 (+ 2193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:13.589713 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:13.591487 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:13.591498 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:13.591675 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:13.591709 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 15:01:13.591799 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 15:01:13.592012 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":938,"mutex_wait_us":63,"thread_start_us":4708,"threads_started":1}
I20260928 15:01:13.593216 17169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:13.581055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38918 (local address 127.31.40.3:40647)
0928 15:01:13.581880 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:13.581888 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:13.584835 (+ 2947us) server_negotiation.cc:407] Connection header received
0928 15:01:13.584986 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:13.584993 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:13.585224 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:13.585411 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:13.586229 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:13.587811 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:13.589666 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:13.591069 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:13.591198 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:13.591429 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:13.592812 (+ 1383us) server_negotiation.cc:299] Negotiation successful
0928 15:01:13.592978 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":452,"threads_started":1}
I20260928 15:01:13.595985 16954 tablet_service.cc:1467] Processing CreateTablet for tablet 4250daea466943329270ce8fee199ec4 (DEFAULT_TABLE table=test-table [id=ff29599c33e74c86a84a1a19cb9d5314]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:01:13.596514 16820 tablet_service.cc:1467] Processing CreateTablet for tablet 4250daea466943329270ce8fee199ec4 (DEFAULT_TABLE table=test-table [id=ff29599c33e74c86a84a1a19cb9d5314]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:01:13.596969 17087 tablet_service.cc:1467] Processing CreateTablet for tablet 4250daea466943329270ce8fee199ec4 (DEFAULT_TABLE table=test-table [id=ff29599c33e74c86a84a1a19cb9d5314]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:01:13.597543 16954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4250daea466943329270ce8fee199ec4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:13.597797 16820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4250daea466943329270ce8fee199ec4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:13.598285 17087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4250daea466943329270ce8fee199ec4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:13.610462 17170 tablet_bootstrap.cc:492] T 4250daea466943329270ce8fee199ec4 P ad269aacd7db4441a4ffd3fd1502cabb: Bootstrap starting.
I20260928 15:01:13.612603 17171 tablet_bootstrap.cc:492] T 4250daea466943329270ce8fee199ec4 P ec57a9abba334534901d01f3ecf1c715: Bootstrap starting.
I20260928 15:01:13.614391 17170 tablet_bootstrap.cc:654] T 4250daea466943329270ce8fee199ec4 P ad269aacd7db4441a4ffd3fd1502cabb: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:13.615293 17172 tablet_bootstrap.cc:492] T 4250daea466943329270ce8fee199ec4 P 10bae2153c1c4d30b26aae53cff4d846: Bootstrap starting.
I20260928 15:01:13.616183 17170 log.cc:824] T 4250daea466943329270ce8fee199ec4 P ad269aacd7db4441a4ffd3fd1502cabb: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:13.618688 17172 tablet_bootstrap.cc:654] T 4250daea466943329270ce8fee199ec4 P 10bae2153c1c4d30b26aae53cff4d846: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:13.618690 17171 tablet_bootstrap.cc:654] T 4250daea466943329270ce8fee199ec4 P ec57a9abba334534901d01f3ecf1c715: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:13.620004 17171 log.cc:824] T 4250daea466943329270ce8fee199ec4 P ec57a9abba334534901d01f3ecf1c715: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:13.620005 17172 log.cc:824] T 4250daea466943329270ce8fee199ec4 P 10bae2153c1c4d30b26aae53cff4d846: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:13.621933 17170 tablet_bootstrap.cc:492] T 4250daea466943329270ce8fee199ec4 P ad269aacd7db4441a4ffd3fd1502cabb: No bootstrap required, opened a new log
I20260928 15:01:13.622242 17170 ts_tablet_manager.cc:1397] T 4250daea466943329270ce8fee199ec4 P ad269aacd7db4441a4ffd3fd1502cabb: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.006s
I20260928 15:01:13.622870 17171 tablet_bootstrap.cc:492] T 4250daea466943329270ce8fee199ec4 P ec57a9abba334534901d01f3ecf1c715: No bootstrap required, opened a new log
I20260928 15:01:13.623134 17171 ts_tablet_manager.cc:1397] T 4250daea466943329270ce8fee199ec4 P ec57a9abba334534901d01f3ecf1c715: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260928 15:01:13.623404 17172 tablet_bootstrap.cc:492] T 4250daea466943329270ce8fee199ec4 P 10bae2153c1c4d30b26aae53cff4d846: No bootstrap required, opened a new log
I20260928 15:01:13.623693 17172 ts_tablet_manager.cc:1397] T 4250daea466943329270ce8fee199ec4 P 10bae2153c1c4d30b26aae53cff4d846: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260928 15:01:13.630842 17171 raft_consensus.cc:348] T 4250daea466943329270ce8fee199ec4 P ec57a9abba334534901d01f3ecf1c715 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec57a9abba334534901d01f3ecf1c715" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 44189 } } peers { permanent_uuid: "10bae2153c1c4d30b26aae53cff4d846" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 40647 } } peers { permanent_uuid: "ad269aacd7db4441a4ffd3fd1502cabb" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 35705 } }
I20260928 15:01:13.631261 17171 raft_consensus.cc:374] T 4250daea466943329270ce8fee199ec4 P ec57a9abba334534901d01f3ecf1c715 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:13.631363 17171 raft_consensus.cc:729] T 4250daea466943329270ce8fee199ec4 P ec57a9abba334534901d01f3ecf1c715 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec57a9abba334534901d01f3ecf1c715, State: Initialized, Role: FOLLOWER
I20260928 15:01:13.632107 17171 consensus_queue.cc:260] T 4250daea466943329270ce8fee199ec4 P ec57a9abba334534901d01f3ecf1c715 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec57a9abba334534901d01f3ecf1c715" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 44189 } } peers { permanent_uuid: "10bae2153c1c4d30b26aae53cff4d846" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 40647 } } peers { permanent_uuid: "ad269aacd7db4441a4ffd3fd1502cabb" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 35705 } }
I20260928 15:01:13.633281 17019 heartbeater.cc:503] Master 127.31.40.62:38363 was elected leader, sending a full tablet report...
I20260928 15:01:13.633328 17170 raft_consensus.cc:348] T 4250daea466943329270ce8fee199ec4 P ad269aacd7db4441a4ffd3fd1502cabb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec57a9abba334534901d01f3ecf1c715" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 44189 } } peers { permanent_uuid: "10bae2153c1c4d30b26aae53cff4d846" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 40647 } } peers { permanent_uuid: "ad269aacd7db4441a4ffd3fd1502cabb" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 35705 } }
I20260928 15:01:13.633761 17170 raft_consensus.cc:374] T 4250daea466943329270ce8fee199ec4 P ad269aacd7db4441a4ffd3fd1502cabb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:13.633844 17170 raft_consensus.cc:729] T 4250daea466943329270ce8fee199ec4 P ad269aacd7db4441a4ffd3fd1502cabb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad269aacd7db4441a4ffd3fd1502cabb, State: Initialized, Role: FOLLOWER
I20260928 15:01:13.633967 17171 ts_tablet_manager.cc:1428] T 4250daea466943329270ce8fee199ec4 P ec57a9abba334534901d01f3ecf1c715: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260928 15:01:13.634537 17170 consensus_queue.cc:260] T 4250daea466943329270ce8fee199ec4 P ad269aacd7db4441a4ffd3fd1502cabb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec57a9abba334534901d01f3ecf1c715" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 44189 } } peers { permanent_uuid: "10bae2153c1c4d30b26aae53cff4d846" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 40647 } } peers { permanent_uuid: "ad269aacd7db4441a4ffd3fd1502cabb" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 35705 } }
I20260928 15:01:13.634452 17172 raft_consensus.cc:348] T 4250daea466943329270ce8fee199ec4 P 10bae2153c1c4d30b26aae53cff4d846 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec57a9abba334534901d01f3ecf1c715" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 44189 } } peers { permanent_uuid: "10bae2153c1c4d30b26aae53cff4d846" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 40647 } } peers { permanent_uuid: "ad269aacd7db4441a4ffd3fd1502cabb" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 35705 } }
I20260928 15:01:13.634891 17172 raft_consensus.cc:374] T 4250daea466943329270ce8fee199ec4 P 10bae2153c1c4d30b26aae53cff4d846 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:13.635001 17172 raft_consensus.cc:729] T 4250daea466943329270ce8fee199ec4 P 10bae2153c1c4d30b26aae53cff4d846 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bae2153c1c4d30b26aae53cff4d846, State: Initialized, Role: FOLLOWER
I20260928 15:01:13.635615 17172 consensus_queue.cc:260] T 4250daea466943329270ce8fee199ec4 P 10bae2153c1c4d30b26aae53cff4d846 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec57a9abba334534901d01f3ecf1c715" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 44189 } } peers { permanent_uuid: "10bae2153c1c4d30b26aae53cff4d846" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 40647 } } peers { permanent_uuid: "ad269aacd7db4441a4ffd3fd1502cabb" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 35705 } }
I20260928 15:01:13.636559 17170 ts_tablet_manager.cc:1428] T 4250daea466943329270ce8fee199ec4 P ad269aacd7db4441a4ffd3fd1502cabb: Time spent starting tablet: real 0.014s user 0.000s sys 0.016s
I20260928 15:01:13.637665 17152 heartbeater.cc:503] Master 127.31.40.62:38363 was elected leader, sending a full tablet report...
I20260928 15:01:13.639072 17172 ts_tablet_manager.cc:1428] T 4250daea466943329270ce8fee199ec4 P 10bae2153c1c4d30b26aae53cff4d846: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
W20260928 15:01:13.725636 16886 tablet.cc:2369] T 4250daea466943329270ce8fee199ec4 P ad269aacd7db4441a4ffd3fd1502cabb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:01:13.738559 17020 tablet.cc:2369] T 4250daea466943329270ce8fee199ec4 P ec57a9abba334534901d01f3ecf1c715: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:01:13.755281 17153 tablet.cc:2369] T 4250daea466943329270ce8fee199ec4 P 10bae2153c1c4d30b26aae53cff4d846: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:01:13.804131 17177 raft_consensus.cc:482] T 4250daea466943329270ce8fee199ec4 P ad269aacd7db4441a4ffd3fd1502cabb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:01:13.804484 17177 raft_consensus.cc:504] T 4250daea466943329270ce8fee199ec4 P ad269aacd7db4441a4ffd3fd1502cabb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec57a9abba334534901d01f3ecf1c715" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 44189 } } peers { permanent_uuid: "10bae2153c1c4d30b26aae53cff4d846" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 40647 } } peers { permanent_uuid: "ad269aacd7db4441a4ffd3fd1502cabb" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 35705 } }
I20260928 15:01:13.806679 17177 leader_election.cc:288] T 4250daea466943329270ce8fee199ec4 P ad269aacd7db4441a4ffd3fd1502cabb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec57a9abba334534901d01f3ecf1c715 (127.31.40.2:44189), 10bae2153c1c4d30b26aae53cff4d846 (127.31.40.3:40647)
I20260928 15:01:13.814561 17165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:13.807191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:50411 (local address 127.31.40.2:44189)
0928 15:01:13.807397 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:13.807402 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:13.808055 (+ 653us) server_negotiation.cc:407] Connection header received
0928 15:01:13.808291 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:13.808297 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:13.808385 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:13.808505 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:13.809190 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:13.810035 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:13.812541 (+ 2506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:13.813612 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:13.813718 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:13.813793 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:13.814217 (+ 424us) server_negotiation.cc:299] Negotiation successful
0928 15:01:13.814333 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 15:01:13.814860 17179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:13.807100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.2:44189 (local address 127.31.40.1:50411)
0928 15:01:13.807927 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:13.807951 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:13.808113 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:13.808532 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:13.808538 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:13.808571 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:13.809039 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:13.809049 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:13.810164 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:13.810186 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:13.812370 (+ 2184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:13.812384 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:13.813840 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:13.813847 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:13.814028 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:13.814070 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 15:01:13.814174 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 15:01:13.814312 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"mutex_wait_us":31,"thread_start_us":442,"threads_started":1}
I20260928 15:01:13.815631 17169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:13.808239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:51865 (local address 127.31.40.3:40647)
0928 15:01:13.808476 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:13.808484 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:13.808976 (+ 492us) server_negotiation.cc:407] Connection header received
0928 15:01:13.809218 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:13.809222 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:13.809279 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:13.809385 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:13.809992 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:13.811113 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:13.813317 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:13.814457 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:13.814583 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:13.814681 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:13.815199 (+ 518us) server_negotiation.cc:299] Negotiation successful
0928 15:01:13.815366 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 15:01:13.815992 16974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4250daea466943329270ce8fee199ec4" candidate_uuid: "ad269aacd7db4441a4ffd3fd1502cabb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec57a9abba334534901d01f3ecf1c715" is_pre_election: true
I20260928 15:01:13.816509 17180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:13.808077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.3:40647 (local address 127.31.40.1:51865)
0928 15:01:13.808783 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:13.808801 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:13.809065 (+ 264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:13.809398 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:13.809403 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:13.809425 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:13.809740 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:13.809747 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:13.811256 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:13.811260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:13.813133 (+ 1873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:13.813148 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:13.814695 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:13.814702 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:13.814974 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:13.815014 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 15:01:13.815931 (+ 917us) client_negotiation.cc:240] Negotiation successful
0928 15:01:13.816059 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"mutex_wait_us":23,"thread_start_us":402,"threads_started":1}
I20260928 15:01:13.816670 16974 raft_consensus.cc:2454] T 4250daea466943329270ce8fee199ec4 P ec57a9abba334534901d01f3ecf1c715 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad269aacd7db4441a4ffd3fd1502cabb in term 0.
I20260928 15:01:13.817364 16773 leader_election.cc:302] T 4250daea466943329270ce8fee199ec4 P ad269aacd7db4441a4ffd3fd1502cabb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad269aacd7db4441a4ffd3fd1502cabb, ec57a9abba334534901d01f3ecf1c715; no voters:
I20260928 15:01:13.817561 17107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4250daea466943329270ce8fee199ec4" candidate_uuid: "ad269aacd7db4441a4ffd3fd1502cabb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bae2153c1c4d30b26aae53cff4d846" is_pre_election: true
I20260928 15:01:13.817783 17177 raft_consensus.cc:2781] T 4250daea466943329270ce8fee199ec4 P ad269aacd7db4441a4ffd3fd1502cabb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:01:13.817974 17177 raft_consensus.cc:482] T 4250daea466943329270ce8fee199ec4 P ad269aacd7db4441a4ffd3fd1502cabb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:01:13.818094 17177 raft_consensus.cc:3037] T 4250daea466943329270ce8fee199ec4 P ad269aacd7db4441a4ffd3fd1502cabb [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:13.818141 17107 raft_consensus.cc:2454] T 4250daea466943329270ce8fee199ec4 P 10bae2153c1c4d30b26aae53cff4d846 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad269aacd7db4441a4ffd3fd1502cabb in term 0.
I20260928 15:01:13.820588 17177 raft_consensus.cc:504] T 4250daea466943329270ce8fee199ec4 P ad269aacd7db4441a4ffd3fd1502cabb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec57a9abba334534901d01f3ecf1c715" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 44189 } } peers { permanent_uuid: "10bae2153c1c4d30b26aae53cff4d846" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 40647 } } peers { permanent_uuid: "ad269aacd7db4441a4ffd3fd1502cabb" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 35705 } }
I20260928 15:01:13.821183 17177 leader_election.cc:288] T 4250daea466943329270ce8fee199ec4 P ad269aacd7db4441a4ffd3fd1502cabb [CANDIDATE]: Term 1 election: Requested vote from peers ec57a9abba334534901d01f3ecf1c715 (127.31.40.2:44189), 10bae2153c1c4d30b26aae53cff4d846 (127.31.40.3:40647)
I20260928 15:01:13.821650 16974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4250daea466943329270ce8fee199ec4" candidate_uuid: "ad269aacd7db4441a4ffd3fd1502cabb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec57a9abba334534901d01f3ecf1c715"
I20260928 15:01:13.821689 17107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4250daea466943329270ce8fee199ec4" candidate_uuid: "ad269aacd7db4441a4ffd3fd1502cabb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bae2153c1c4d30b26aae53cff4d846"
I20260928 15:01:13.821898 17107 raft_consensus.cc:3037] T 4250daea466943329270ce8fee199ec4 P 10bae2153c1c4d30b26aae53cff4d846 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:13.821931 16974 raft_consensus.cc:3037] T 4250daea466943329270ce8fee199ec4 P ec57a9abba334534901d01f3ecf1c715 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:13.825146 16974 raft_consensus.cc:2454] T 4250daea466943329270ce8fee199ec4 P ec57a9abba334534901d01f3ecf1c715 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad269aacd7db4441a4ffd3fd1502cabb in term 1.
I20260928 15:01:13.825146 17107 raft_consensus.cc:2454] T 4250daea466943329270ce8fee199ec4 P 10bae2153c1c4d30b26aae53cff4d846 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad269aacd7db4441a4ffd3fd1502cabb in term 1.
I20260928 15:01:13.825605 16773 leader_election.cc:302] T 4250daea466943329270ce8fee199ec4 P ad269aacd7db4441a4ffd3fd1502cabb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad269aacd7db4441a4ffd3fd1502cabb, ec57a9abba334534901d01f3ecf1c715; no voters:
I20260928 15:01:13.825963 17177 raft_consensus.cc:2781] T 4250daea466943329270ce8fee199ec4 P ad269aacd7db4441a4ffd3fd1502cabb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:01:13.826531 17177 raft_consensus.cc:686] T 4250daea466943329270ce8fee199ec4 P ad269aacd7db4441a4ffd3fd1502cabb [term 1 LEADER]: Becoming Leader. State: Replica: ad269aacd7db4441a4ffd3fd1502cabb, State: Running, Role: LEADER
I20260928 15:01:13.827451 17177 consensus_queue.cc:237] T 4250daea466943329270ce8fee199ec4 P ad269aacd7db4441a4ffd3fd1502cabb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec57a9abba334534901d01f3ecf1c715" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 44189 } } peers { permanent_uuid: "10bae2153c1c4d30b26aae53cff4d846" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 40647 } } peers { permanent_uuid: "ad269aacd7db4441a4ffd3fd1502cabb" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 35705 } }
I20260928 15:01:13.833210 16695 catalog_manager.cc:5495] T 4250daea466943329270ce8fee199ec4 P ad269aacd7db4441a4ffd3fd1502cabb reported cstate change: term changed from 0 to 1, leader changed from <none> to ad269aacd7db4441a4ffd3fd1502cabb (127.31.40.1). New cstate: current_term: 1 leader_uuid: "ad269aacd7db4441a4ffd3fd1502cabb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec57a9abba334534901d01f3ecf1c715" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 44189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10bae2153c1c4d30b26aae53cff4d846" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 40647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad269aacd7db4441a4ffd3fd1502cabb" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 35705 } health_report { overall_health: HEALTHY } } }
I20260928 15:01:13.887454 17168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:13.881676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50882 (local address 127.31.40.1:35705)
0928 15:01:13.881913 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:13.881918 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:13.882057 (+ 139us) server_negotiation.cc:407] Connection header received
0928 15:01:13.882259 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:13.882263 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:13.882344 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:13.882471 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:01:13.882853 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:13.883596 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:13.885292 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:13.885687 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:13.885783 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:13.886412 (+ 629us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:01:13.886947 (+ 535us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:01:13.887055 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:13.887178 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 15:01:13.887247 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 15:01:13.896186 17107 raft_consensus.cc:1261] T 4250daea466943329270ce8fee199ec4 P 10bae2153c1c4d30b26aae53cff4d846 [term 1 FOLLOWER]: Refusing update from remote peer ad269aacd7db4441a4ffd3fd1502cabb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:01:13.896217 16974 raft_consensus.cc:1261] T 4250daea466943329270ce8fee199ec4 P ec57a9abba334534901d01f3ecf1c715 [term 1 FOLLOWER]: Refusing update from remote peer ad269aacd7db4441a4ffd3fd1502cabb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:01:13.897300 17181 consensus_queue.cc:1035] T 4250daea466943329270ce8fee199ec4 P ad269aacd7db4441a4ffd3fd1502cabb [LEADER]: Connected to new peer: Peer: permanent_uuid: "10bae2153c1c4d30b26aae53cff4d846" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 40647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:01:13.897692 17177 consensus_queue.cc:1035] T 4250daea466943329270ce8fee199ec4 P ad269aacd7db4441a4ffd3fd1502cabb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec57a9abba334534901d01f3ecf1c715" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 44189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:01:13.927273 16694 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55868:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:01:13.927603 16694 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55868:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:01:13.931530 16694 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 54ed0dec12464877825c48ee85b5b65c: Sending DeleteTablet for 3 replicas of tablet 4250daea466943329270ce8fee199ec4
I20260928 15:01:13.932817 16954 tablet_service.cc:1514] Processing DeleteTablet for tablet 4250daea466943329270ce8fee199ec4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:01:13 UTC) from {username='kudu', principal='kudu/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:49204
I20260928 15:01:13.933000 17087 tablet_service.cc:1514] Processing DeleteTablet for tablet 4250daea466943329270ce8fee199ec4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:01:13 UTC) from {username='kudu', principal='kudu/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:38918
I20260928 15:01:13.933831 17194 tablet_replica.cc:330] T 4250daea466943329270ce8fee199ec4 P ec57a9abba334534901d01f3ecf1c715: stopping tablet replica
I20260928 15:01:13.934094 16820 tablet_service.cc:1514] Processing DeleteTablet for tablet 4250daea466943329270ce8fee199ec4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:01:13 UTC) from {username='kudu', principal='kudu/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:50866
I20260928 15:01:13.934450 17194 raft_consensus.cc:2229] T 4250daea466943329270ce8fee199ec4 P ec57a9abba334534901d01f3ecf1c715 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:01:13.934521 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 16757
I20260928 15:01:13.935072 17194 raft_consensus.cc:2258] T 4250daea466943329270ce8fee199ec4 P ec57a9abba334534901d01f3ecf1c715 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:01:13.937765 17194 ts_tablet_manager.cc:1905] T 4250daea466943329270ce8fee199ec4 P ec57a9abba334534901d01f3ecf1c715: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:01:13.947912 17194 ts_tablet_manager.cc:1918] T 4250daea466943329270ce8fee199ec4 P ec57a9abba334534901d01f3ecf1c715: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:01:13.948251 17194 log.cc:1196] T 4250daea466943329270ce8fee199ec4 P ec57a9abba334534901d01f3ecf1c715: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/ts-1/wal/wals/4250daea466943329270ce8fee199ec4
I20260928 15:01:13.948956 17194 ts_tablet_manager.cc:1939] T 4250daea466943329270ce8fee199ec4 P ec57a9abba334534901d01f3ecf1c715: Deleting consensus metadata
I20260928 15:01:13.949810 17195 tablet_replica.cc:330] T 4250daea466943329270ce8fee199ec4 P 10bae2153c1c4d30b26aae53cff4d846: stopping tablet replica
I20260928 15:01:13.950279 17195 raft_consensus.cc:2229] T 4250daea466943329270ce8fee199ec4 P 10bae2153c1c4d30b26aae53cff4d846 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:01:13.950822 17195 raft_consensus.cc:2258] T 4250daea466943329270ce8fee199ec4 P 10bae2153c1c4d30b26aae53cff4d846 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:01:13.951555 16680 catalog_manager.cc:4841] TS ec57a9abba334534901d01f3ecf1c715 (127.31.40.2:44189): tablet 4250daea466943329270ce8fee199ec4 (table test-table [id=ff29599c33e74c86a84a1a19cb9d5314]) successfully deleted
I20260928 15:01:13.953253 17196 tablet_replica.cc:330] T 4250daea466943329270ce8fee199ec4 P ad269aacd7db4441a4ffd3fd1502cabb: stopping tablet replica
I20260928 15:01:13.954144 17195 ts_tablet_manager.cc:1905] T 4250daea466943329270ce8fee199ec4 P 10bae2153c1c4d30b26aae53cff4d846: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:01:13.954174 17196 raft_consensus.cc:2229] T 4250daea466943329270ce8fee199ec4 P ad269aacd7db4441a4ffd3fd1502cabb [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:01:13.954934 17196 raft_consensus.cc:2258] T 4250daea466943329270ce8fee199ec4 P ad269aacd7db4441a4ffd3fd1502cabb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:01:13.957700 17196 ts_tablet_manager.cc:1905] T 4250daea466943329270ce8fee199ec4 P ad269aacd7db4441a4ffd3fd1502cabb: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 15:01:13 dist-test-slave-j968 krb5kdc[16651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.1@KRBTEST.COM
I20260928 15:01:13.961794 17195 ts_tablet_manager.cc:1918] T 4250daea466943329270ce8fee199ec4 P 10bae2153c1c4d30b26aae53cff4d846: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:01:13.961982 17195 log.cc:1196] T 4250daea466943329270ce8fee199ec4 P 10bae2153c1c4d30b26aae53cff4d846: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/ts-2/wal/wals/4250daea466943329270ce8fee199ec4
I20260928 15:01:13.962550 17195 ts_tablet_manager.cc:1939] T 4250daea466943329270ce8fee199ec4 P 10bae2153c1c4d30b26aae53cff4d846: Deleting consensus metadata
I20260928 15:01:13.964507 16680 catalog_manager.cc:4841] TS 10bae2153c1c4d30b26aae53cff4d846 (127.31.40.3:40647): tablet 4250daea466943329270ce8fee199ec4 (table test-table [id=ff29599c33e74c86a84a1a19cb9d5314]) successfully deleted
I20260928 15:01:13.966679 17196 ts_tablet_manager.cc:1918] T 4250daea466943329270ce8fee199ec4 P ad269aacd7db4441a4ffd3fd1502cabb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:01:13.966861 17196 log.cc:1196] T 4250daea466943329270ce8fee199ec4 P ad269aacd7db4441a4ffd3fd1502cabb: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607365362981-31904-0/minicluster-data/ts-0/wal/wals/4250daea466943329270ce8fee199ec4
I20260928 15:01:13.967396 17196 ts_tablet_manager.cc:1939] T 4250daea466943329270ce8fee199ec4 P ad269aacd7db4441a4ffd3fd1502cabb: Deleting consensus metadata
I20260928 15:01:13.969254 17168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:13.936265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50898 (local address 127.31.40.1:35705)
0928 15:01:13.936484 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:13.936490 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:13.946532 (+ 10042us) server_negotiation.cc:407] Connection header received
0928 15:01:13.947042 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:13.947048 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:13.947136 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:13.947207 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:13.951986 (+ 4779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:13.953322 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:13.955286 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:13.956105 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:13.956229 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:13.961999 (+ 5770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:13.962016 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:13.962034 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:13.962091 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:13.965994 (+ 3903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:13.966749 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:13.966756 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:13.966759 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:13.966979 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:13.967453 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:13.967459 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:13.967462 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:13.967913 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:13.968256 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:13.968570 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 15:01:13.968859 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 15:01:13.969224 16683 catalog_manager.cc:4841] TS ad269aacd7db4441a4ffd3fd1502cabb (127.31.40.1:35705): tablet 4250daea466943329270ce8fee199ec4 (table test-table [id=ff29599c33e74c86a84a1a19cb9d5314]) successfully deleted
I20260928 15:01:13.969990 16880 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:01:14.072024 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 16757
I20260928 15:01:14.091753 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 16891
Sep 28 15:01:14 dist-test-slave-j968 krb5kdc[16651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.2@KRBTEST.COM
I20260928 15:01:14.107069 17165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:14.092594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49206 (local address 127.31.40.2:44189)
0928 15:01:14.092848 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:14.092856 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:14.092883 (+ 27us) server_negotiation.cc:407] Connection header received
0928 15:01:14.093015 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:14.093019 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:14.093083 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:14.093212 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:14.094494 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:14.095221 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:14.096660 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:14.097089 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:14.097192 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:14.101374 (+ 4182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:14.101390 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:14.101406 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:14.101439 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:14.104415 (+ 2976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:14.105054 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:14.105062 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:14.105064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:14.105178 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:14.105532 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:14.105536 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:14.105538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:14.105944 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:14.106175 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:14.106424 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 15:01:14.106589 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 15:01:14.107787 17014 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:01:14.232314 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 16891
I20260928 15:01:14.257368 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 17024
Sep 28 15:01:14 dist-test-slave-j968 krb5kdc[16651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.3@KRBTEST.COM
I20260928 15:01:14.274044 17169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:14.258310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38934 (local address 127.31.40.3:40647)
0928 15:01:14.258486 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:14.258490 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:14.258563 (+ 73us) server_negotiation.cc:407] Connection header received
0928 15:01:14.258750 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:14.258753 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:14.258825 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:14.258918 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:14.260832 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:14.261545 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:14.263582 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:14.264124 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:14.264218 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:14.268591 (+ 4373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:14.268607 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:14.268623 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:14.268654 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:14.271374 (+ 2720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:14.272024 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:14.272031 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:14.272069 (+ 38us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:14.272231 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:14.272556 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:14.272559 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:14.272561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:14.272904 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:14.273128 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:14.273520 (+ 392us) server_negotiation.cc:299] Negotiation successful
0928 15:01:14.273676 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 15:01:14.274773 17147 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:01:14.364202 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 17024
I20260928 15:01:14.382164 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 16664
I20260928 15:01:14.383313 16726 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:01:14.496997 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 16664
2026-09-28T15:01:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4500 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T15:01:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:01:14Z Disabled control of system clock
I20260928 15:01:14.581584 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:42745
--webserver_interface=127.31.40.62
--webserver_port=0
--builtin_ntp_servers=127.31.40.20:44025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.40.62:42745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 15:01:14.923619 17205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:14.923988 17205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:14.924081 17205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:14.931473 17205 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:01:14.931582 17205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:14.931627 17205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:14.931682 17205 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:01:14.931733 17205 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:01:14.941259 17205 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:44025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.40.62:42745
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:42745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:01:14.942535 17205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:14.944419 17205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:14.956423 17211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:14.956451 17210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:14.956429 17213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:14.957275 17205 server_base.cc:1034] running on GCE node
I20260928 15:01:14.958048 17205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:14.959707 17205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:14.960949 17205 hybrid_clock.cc:648] HybridClock initialized: now 1790607674960878 us; error 67 us; skew 500 ppm
I20260928 15:01:14.961333 17205 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:01:14.963620 17205 webserver.cc:460] Webserver started at http://127.31.40.62:44637/ using document root <none> and password file <none>
I20260928 15:01:14.964486 17205 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:14.964599 17205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:14.964926 17205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:14.967798 17205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/master-0/data/instance:
uuid: "6720069b68af4fcfbc46d12a737cfe62"
format_stamp: "Formatted at 2026-09-28 15:01:14 on dist-test-slave-j968"
server_key: "1ddd47a34f91a7d7a6c45e68fd589adb"
server_key_iv: "ef7ec70b33c2d4412a27244c5f8e3bda"
server_key_version: "encryptionkey@0"
I20260928 15:01:14.968691 17205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/master-0/wal/instance:
uuid: "6720069b68af4fcfbc46d12a737cfe62"
format_stamp: "Formatted at 2026-09-28 15:01:14 on dist-test-slave-j968"
server_key: "1ddd47a34f91a7d7a6c45e68fd589adb"
server_key_iv: "ef7ec70b33c2d4412a27244c5f8e3bda"
server_key_version: "encryptionkey@0"
I20260928 15:01:14.973747 17205 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 15:01:14.977421 17219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:14.978809 17205 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 15:01:14.979038 17205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/master-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/master-0/wal
uuid: "6720069b68af4fcfbc46d12a737cfe62"
format_stamp: "Formatted at 2026-09-28 15:01:14 on dist-test-slave-j968"
server_key: "1ddd47a34f91a7d7a6c45e68fd589adb"
server_key_iv: "ef7ec70b33c2d4412a27244c5f8e3bda"
server_key_version: "encryptionkey@0"
I20260928 15:01:14.979259 17205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:14.995762 17205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:14.996594 17205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:14.996883 17205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:15.015530 17205 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.62:42745
I20260928 15:01:15.015551 17270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.62:42745 every 8 connection(s)
I20260928 15:01:15.016985 17205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
I20260928 15:01:15.020715 17271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:15.025511 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 17205
I20260928 15:01:15.025676 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/master-0/wal/instance
I20260928 15:01:15.026211 31904 external_mini_cluster.cc:1428] Setting key 37f76d8965bb8dfd8cee7442d772b0f1
I20260928 15:01:15.032722 17271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6720069b68af4fcfbc46d12a737cfe62: Bootstrap starting.
I20260928 15:01:15.036405 17271 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6720069b68af4fcfbc46d12a737cfe62: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:15.037557 17271 log.cc:824] T 00000000000000000000000000000000 P 6720069b68af4fcfbc46d12a737cfe62: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:15.040964 17271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6720069b68af4fcfbc46d12a737cfe62: No bootstrap required, opened a new log
I20260928 15:01:15.042325 17274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:15.028311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37236 (local address 127.31.40.62:42745)
0928 15:01:15.029449 (+ 1138us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:15.029464 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:15.029517 (+ 53us) server_negotiation.cc:407] Connection header received
0928 15:01:15.031964 (+ 2447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:15.031991 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:15.032542 (+ 551us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:15.032962 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:15.033487 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:15.035478 (+ 1991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:15.037021 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:15.038108 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:15.038300 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:15.039599 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:15.039609 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:15.039625 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:01:15.040351 (+ 726us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:15.040449 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:01:15.040475 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:15.040590 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:15.040778 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 15:01:15.041105 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":357,"threads_started":1}
I20260928 15:01:15.049263 17271 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6720069b68af4fcfbc46d12a737cfe62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6720069b68af4fcfbc46d12a737cfe62" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 42745 } }
I20260928 15:01:15.049651 17271 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6720069b68af4fcfbc46d12a737cfe62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:15.049726 17271 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6720069b68af4fcfbc46d12a737cfe62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6720069b68af4fcfbc46d12a737cfe62, State: Initialized, Role: FOLLOWER
I20260928 15:01:15.050264 17271 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6720069b68af4fcfbc46d12a737cfe62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6720069b68af4fcfbc46d12a737cfe62" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 42745 } }
I20260928 15:01:15.050472 17271 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6720069b68af4fcfbc46d12a737cfe62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:01:15.050619 17271 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6720069b68af4fcfbc46d12a737cfe62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:01:15.050784 17271 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6720069b68af4fcfbc46d12a737cfe62 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:15.053066 17271 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6720069b68af4fcfbc46d12a737cfe62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6720069b68af4fcfbc46d12a737cfe62" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 42745 } }
I20260928 15:01:15.053540 17271 leader_election.cc:302] T 00000000000000000000000000000000 P 6720069b68af4fcfbc46d12a737cfe62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6720069b68af4fcfbc46d12a737cfe62; no voters:
I20260928 15:01:15.054086 17271 leader_election.cc:288] T 00000000000000000000000000000000 P 6720069b68af4fcfbc46d12a737cfe62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:01:15.054241 17276 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6720069b68af4fcfbc46d12a737cfe62 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:01:15.055024 17276 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6720069b68af4fcfbc46d12a737cfe62 [term 1 LEADER]: Becoming Leader. State: Replica: 6720069b68af4fcfbc46d12a737cfe62, State: Running, Role: LEADER
I20260928 15:01:15.055562 17276 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6720069b68af4fcfbc46d12a737cfe62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6720069b68af4fcfbc46d12a737cfe62" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 42745 } }
I20260928 15:01:15.056334 17271 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6720069b68af4fcfbc46d12a737cfe62 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:01:15.058182 17277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6720069b68af4fcfbc46d12a737cfe62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6720069b68af4fcfbc46d12a737cfe62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6720069b68af4fcfbc46d12a737cfe62" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 42745 } } }
I20260928 15:01:15.058508 17277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6720069b68af4fcfbc46d12a737cfe62 [sys.catalog]: This master's current role is: LEADER
I20260928 15:01:15.059819 17278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6720069b68af4fcfbc46d12a737cfe62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6720069b68af4fcfbc46d12a737cfe62. Latest consensus state: current_term: 1 leader_uuid: "6720069b68af4fcfbc46d12a737cfe62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6720069b68af4fcfbc46d12a737cfe62" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 42745 } } }
I20260928 15:01:15.060201 17278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6720069b68af4fcfbc46d12a737cfe62 [sys.catalog]: This master's current role is: LEADER
I20260928 15:01:15.062110 17282 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:01:15.067694 17282 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:01:15.076448 17282 catalog_manager.cc:1347] Generated new cluster ID: b24ff57770f943089453fd0c900c6ccb
I20260928 15:01:15.076582 17282 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:01:15.099620 17282 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:01:15.100998 17282 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:01:15.109783 17282 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6720069b68af4fcfbc46d12a737cfe62: Generated new TSK 0
I20260928 15:01:15.110697 17282 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:01:15.116446 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.1:0
--local_ip_for_outbound_sockets=127.31.40.1
--webserver_interface=127.31.40.1
--webserver_port=0
--tserver_master_addrs=127.31.40.62:42745
--builtin_ntp_servers=127.31.40.20:44025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 15:01:15.477200 17295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:15.477509 17295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:15.477596 17295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:15.485271 17295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:15.485378 17295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:15.485487 17295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.1
I20260928 15:01:15.494848 17295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:44025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.40.1
--webserver_port=0
--tserver_master_addrs=127.31.40.62:42745
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.1
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:01:15.496178 17295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:15.497931 17295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:15.511740 17301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:15.511981 17300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:15.511744 17303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:15.512614 17295 server_base.cc:1034] running on GCE node
I20260928 15:01:15.513226 17295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:15.514257 17295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:15.515496 17295 hybrid_clock.cc:648] HybridClock initialized: now 1790607675515446 us; error 57 us; skew 500 ppm
I20260928 15:01:15.515839 17295 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:01:15.518126 17295 webserver.cc:460] Webserver started at http://127.31.40.1:45207/ using document root <none> and password file <none>
I20260928 15:01:15.518838 17295 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:15.518972 17295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:15.519295 17295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:15.522245 17295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/ts-0/data/instance:
uuid: "43aa27a441984beab35828e1abfd3eeb"
format_stamp: "Formatted at 2026-09-28 15:01:15 on dist-test-slave-j968"
server_key: "d117adcb30e905300f4849276df29f7a"
server_key_iv: "c40541d9b72c899b3bce4c3daf9e9215"
server_key_version: "encryptionkey@0"
I20260928 15:01:15.523036 17295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/ts-0/wal/instance:
uuid: "43aa27a441984beab35828e1abfd3eeb"
format_stamp: "Formatted at 2026-09-28 15:01:15 on dist-test-slave-j968"
server_key: "d117adcb30e905300f4849276df29f7a"
server_key_iv: "c40541d9b72c899b3bce4c3daf9e9215"
server_key_version: "encryptionkey@0"
I20260928 15:01:15.528697 17295 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 15:01:15.532245 17309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:15.533525 17295 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 15:01:15.533715 17295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/ts-0/wal
uuid: "43aa27a441984beab35828e1abfd3eeb"
format_stamp: "Formatted at 2026-09-28 15:01:15 on dist-test-slave-j968"
server_key: "d117adcb30e905300f4849276df29f7a"
server_key_iv: "c40541d9b72c899b3bce4c3daf9e9215"
server_key_version: "encryptionkey@0"
I20260928 15:01:15.533941 17295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:15.545951 17295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:15.546788 17295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:15.547103 17295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:15.548199 17295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:01:15.549721 17295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:01:15.549856 17295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:15.549988 17295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:01:15.550074 17295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:15.586064 17295 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.1:39763
I20260928 15:01:15.586115 17421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.1:39763 every 8 connection(s)
I20260928 15:01:15.587474 17295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
I20260928 15:01:15.595505 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 17295
I20260928 15:01:15.595690 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/ts-0/wal/instance
I20260928 15:01:15.596235 31904 external_mini_cluster.cc:1428] Setting key fb3d87e11ac32f1a2562630d47d8b550
I20260928 15:01:15.599234 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.2:0
--local_ip_for_outbound_sockets=127.31.40.2
--webserver_interface=127.31.40.2
--webserver_port=0
--tserver_master_addrs=127.31.40.62:42745
--builtin_ntp_servers=127.31.40.20:44025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 15:01:15.600883 17424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:15.590116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:36517 (local address 127.31.40.62:42745)
0928 15:01:15.590564 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:15.590568 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:15.591685 (+ 1117us) server_negotiation.cc:407] Connection header received
0928 15:01:15.593498 (+ 1813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:15.593504 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:15.593592 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:15.593710 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:15.594786 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:15.595805 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:15.597758 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:15.598325 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:15.598451 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:15.599837 (+ 1386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:15.599847 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:15.599855 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:01:15.599908 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:15.599963 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:01:15.599983 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:15.600078 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:15.600322 (+ 244us) server_negotiation.cc:299] Negotiation successful
0928 15:01:15.600430 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":244,"threads_started":1}
I20260928 15:01:15.601707 17425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:15.590670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:42745 (local address 127.31.40.1:36517)
0928 15:01:15.591540 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:15.591576 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:15.593218 (+ 1642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:15.593958 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:15.593966 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:15.594054 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:01:15.594619 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:15.594634 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:15.595990 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:15.595994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:15.597622 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:15.597636 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:15.598527 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:15.598532 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:15.598736 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:15.599535 (+ 799us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:01:15.599538 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:01:15.599578 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:01:15.599651 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:01:15.600100 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:01:15.600110 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:01:15.600117 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 15:01:15.600294 (+ 177us) client_negotiation.cc:240] Negotiation successful
0928 15:01:15.600588 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":276,"threads_started":1}
I20260928 15:01:15.603228 17422 heartbeater.cc:347] Connected to a master server at 127.31.40.62:42745
I20260928 15:01:15.603538 17422 heartbeater.cc:464] Registering TS with master...
I20260928 15:01:15.604334 17422 heartbeater.cc:511] Master 127.31.40.62:42745 requested a full tablet report, sending...
I20260928 15:01:15.606426 17236 ts_manager.cc:194] Registered new tserver with Master: 43aa27a441984beab35828e1abfd3eeb (127.31.40.1:39763)
I20260928 15:01:15.608080 17236 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.40.1:36517
W20260928 15:01:15.949030 17426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:15.949306 17426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:15.949399 17426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:15.956960 17426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:15.957058 17426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:15.957165 17426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.2
I20260928 15:01:15.966378 17426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:44025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.40.2
--webserver_port=0
--tserver_master_addrs=127.31.40.62:42745
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.2
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:01:15.967614 17426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:15.969424 17426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:15.982676 17434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:15.982750 17432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:15.982723 17431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:15.983795 17426 server_base.cc:1034] running on GCE node
I20260928 15:01:15.984519 17426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:15.985436 17426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:15.986644 17426 hybrid_clock.cc:648] HybridClock initialized: now 1790607675986595 us; error 35 us; skew 500 ppm
I20260928 15:01:15.986964 17426 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:01:15.989212 17426 webserver.cc:460] Webserver started at http://127.31.40.2:41449/ using document root <none> and password file <none>
I20260928 15:01:15.989894 17426 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:15.990008 17426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:15.990329 17426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:15.993299 17426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/ts-1/data/instance:
uuid: "e7f64b778c544179b8a81c5ab424a47d"
format_stamp: "Formatted at 2026-09-28 15:01:15 on dist-test-slave-j968"
server_key: "87a86c92cec0078a2f7c3dc20aa2ffa6"
server_key_iv: "2fc7760187b7a8c7317e90e827b35d1c"
server_key_version: "encryptionkey@0"
I20260928 15:01:15.994078 17426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/ts-1/wal/instance:
uuid: "e7f64b778c544179b8a81c5ab424a47d"
format_stamp: "Formatted at 2026-09-28 15:01:15 on dist-test-slave-j968"
server_key: "87a86c92cec0078a2f7c3dc20aa2ffa6"
server_key_iv: "2fc7760187b7a8c7317e90e827b35d1c"
server_key_version: "encryptionkey@0"
I20260928 15:01:15.999539 17426 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260928 15:01:16.003147 17440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:16.004635 17426 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:01:16.004812 17426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/ts-1/wal
uuid: "e7f64b778c544179b8a81c5ab424a47d"
format_stamp: "Formatted at 2026-09-28 15:01:15 on dist-test-slave-j968"
server_key: "87a86c92cec0078a2f7c3dc20aa2ffa6"
server_key_iv: "2fc7760187b7a8c7317e90e827b35d1c"
server_key_version: "encryptionkey@0"
I20260928 15:01:16.005031 17426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:16.026446 17426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:16.027246 17426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:16.027532 17426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:16.028664 17426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:01:16.030144 17426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:01:16.030267 17426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:16.030422 17426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:01:16.030512 17426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:16.063844 17426 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.2:38827
I20260928 15:01:16.064002 17552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.2:38827 every 8 connection(s)
I20260928 15:01:16.065222 17426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
I20260928 15:01:16.065959 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 17426
I20260928 15:01:16.066164 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/ts-1/wal/instance
I20260928 15:01:16.066790 31904 external_mini_cluster.cc:1428] Setting key ad8246b8e4ea2da0055617e82088d58c
I20260928 15:01:16.070024 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.3:0
--local_ip_for_outbound_sockets=127.31.40.3
--webserver_interface=127.31.40.3
--webserver_port=0
--tserver_master_addrs=127.31.40.62:42745
--builtin_ntp_servers=127.31.40.20:44025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 15:01:16.080888 17424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:16.068129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:59437 (local address 127.31.40.62:42745)
0928 15:01:16.068346 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:16.068353 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:16.070181 (+ 1828us) server_negotiation.cc:407] Connection header received
0928 15:01:16.073057 (+ 2876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:16.073061 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:16.073127 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:16.073212 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:16.074617 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:16.075341 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:16.077875 (+ 2534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:16.078279 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:16.078381 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:16.080069 (+ 1688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:16.080082 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:16.080086 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:01:16.080106 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:16.080138 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:01:16.080150 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:16.080240 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:16.080480 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 15:01:16.080582 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 15:01:16.081967 17555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:16.068877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:42745 (local address 127.31.40.2:59437)
0928 15:01:16.070001 (+ 1124us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:16.070060 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:16.072742 (+ 2682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:16.073472 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:16.073486 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:16.073615 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:01:16.074451 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:16.074477 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:16.075503 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:16.075512 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:16.077693 (+ 2181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:16.077711 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:16.078509 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:16.078519 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:16.078860 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:16.079676 (+ 816us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:01:16.079678 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:01:16.079718 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:01:16.079803 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:01:16.080255 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:01:16.080267 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:01:16.080277 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 15:01:16.080455 (+ 178us) client_negotiation.cc:240] Negotiation successful
0928 15:01:16.080876 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":317,"threads_started":1}
I20260928 15:01:16.083487 17553 heartbeater.cc:347] Connected to a master server at 127.31.40.62:42745
I20260928 15:01:16.083735 17553 heartbeater.cc:464] Registering TS with master...
I20260928 15:01:16.084666 17553 heartbeater.cc:511] Master 127.31.40.62:42745 requested a full tablet report, sending...
I20260928 15:01:16.086275 17236 ts_manager.cc:194] Registered new tserver with Master: e7f64b778c544179b8a81c5ab424a47d (127.31.40.2:38827)
I20260928 15:01:16.087096 17236 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.40.2:59437
W20260928 15:01:16.409204 17556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:16.409492 17556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:16.409658 17556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:16.417466 17556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:16.417582 17556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:16.417775 17556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.3
I20260928 15:01:16.427160 17556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:44025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.40.3
--webserver_port=0
--tserver_master_addrs=127.31.40.62:42745
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.3
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:01:16.428539 17556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:16.430362 17556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:16.443928 17562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:16.443951 17561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:16.443996 17564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:16.444830 17556 server_base.cc:1034] running on GCE node
I20260928 15:01:16.445439 17556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:16.446420 17556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:16.447697 17556 hybrid_clock.cc:648] HybridClock initialized: now 1790607676447630 us; error 52 us; skew 500 ppm
I20260928 15:01:16.448134 17556 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:01:16.450436 17556 webserver.cc:460] Webserver started at http://127.31.40.3:46705/ using document root <none> and password file <none>
I20260928 15:01:16.451149 17556 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:16.451292 17556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:16.451610 17556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:16.454547 17556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/ts-2/data/instance:
uuid: "4f2ab89e862240cb9295ec34283cca12"
format_stamp: "Formatted at 2026-09-28 15:01:16 on dist-test-slave-j968"
server_key: "af709adc809abaf7a65b91929a7d8bc9"
server_key_iv: "4c5cb6ef1e614876baabc2e67cd4e6eb"
server_key_version: "encryptionkey@0"
I20260928 15:01:16.455353 17556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/ts-2/wal/instance:
uuid: "4f2ab89e862240cb9295ec34283cca12"
format_stamp: "Formatted at 2026-09-28 15:01:16 on dist-test-slave-j968"
server_key: "af709adc809abaf7a65b91929a7d8bc9"
server_key_iv: "4c5cb6ef1e614876baabc2e67cd4e6eb"
server_key_version: "encryptionkey@0"
I20260928 15:01:16.460518 17556 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260928 15:01:16.464141 17570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:16.465517 17556 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:01:16.465696 17556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/ts-2/wal
uuid: "4f2ab89e862240cb9295ec34283cca12"
format_stamp: "Formatted at 2026-09-28 15:01:16 on dist-test-slave-j968"
server_key: "af709adc809abaf7a65b91929a7d8bc9"
server_key_iv: "4c5cb6ef1e614876baabc2e67cd4e6eb"
server_key_version: "encryptionkey@0"
I20260928 15:01:16.465916 17556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:16.492063 17556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:16.492949 17556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:16.493247 17556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:16.494292 17556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:01:16.495800 17556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:01:16.495950 17556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 15:01:16.496083 17556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:01:16.496171 17556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:16.530095 17556 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.3:36395
I20260928 15:01:16.530148 17682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.3:36395 every 8 connection(s)
I20260928 15:01:16.531522 17556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
I20260928 15:01:16.538915 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 17556
I20260928 15:01:16.539104 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/ts-2/wal/instance
I20260928 15:01:16.539589 31904 external_mini_cluster.cc:1428] Setting key 855ab0f6aab090dd8c71bbb8b057a1e3
I20260928 15:01:16.545701 17424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:16.534390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:37447 (local address 127.31.40.62:42745)
0928 15:01:16.534615 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:16.534619 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:16.536055 (+ 1436us) server_negotiation.cc:407] Connection header received
0928 15:01:16.538041 (+ 1986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:16.538046 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:16.538144 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:16.538249 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:16.539612 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:16.540658 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:16.542983 (+ 2325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:16.543425 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:16.543525 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:16.544962 (+ 1437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:16.544967 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:16.544970 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:01:16.544987 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:16.545019 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:01:16.545030 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:16.545102 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:16.545380 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 15:01:16.545467 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260928 15:01:16.546756 17685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:16.534889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:42745 (local address 127.31.40.3:37447)
0928 15:01:16.535839 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:16.535945 (+ 106us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:16.537681 (+ 1736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:16.538499 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:16.538508 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:16.538597 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:01:16.539419 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:16.539441 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:16.540830 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:16.540836 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:16.542819 (+ 1983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:16.542834 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:16.543631 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:16.543636 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:16.543964 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:16.544648 (+ 684us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:01:16.544651 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:01:16.544696 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:01:16.544781 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:01:16.545145 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:01:16.545154 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:01:16.545160 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 15:01:16.545348 (+ 188us) client_negotiation.cc:240] Negotiation successful
0928 15:01:16.545649 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":303,"threads_started":1}
I20260928 15:01:16.548208 17683 heartbeater.cc:347] Connected to a master server at 127.31.40.62:42745
I20260928 15:01:16.548501 17683 heartbeater.cc:464] Registering TS with master...
I20260928 15:01:16.549201 17683 heartbeater.cc:511] Master 127.31.40.62:42745 requested a full tablet report, sending...
I20260928 15:01:16.550685 17236 ts_manager.cc:194] Registered new tserver with Master: 4f2ab89e862240cb9295ec34283cca12 (127.31.40.3:36395)
I20260928 15:01:16.551501 17236 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.40.3:37447
I20260928 15:01:16.555373 31904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:01:16.567124 17424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:16.559914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37244 (local address 127.31.40.62:42745)
0928 15:01:16.560178 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:16.560186 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:16.560492 (+ 306us) server_negotiation.cc:407] Connection header received
0928 15:01:16.560697 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:16.560703 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:16.560791 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:16.560896 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:16.561415 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:16.562217 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:16.564480 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:16.565114 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:16.565250 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:16.566458 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:16.566463 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:16.566466 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:01:16.566484 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:16.566517 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:01:16.566528 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:16.566597 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:16.566692 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 15:01:16.566767 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 15:01:16.571604 17236 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37244:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:01:16.578655 17236 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:01:16.602738 17695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:16.591955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:39763 (local address 127.0.0.1:57840)
0928 15:01:16.592940 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:16.592995 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:16.593162 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:16.594232 (+ 1070us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:16.594240 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:16.594276 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:16.594703 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:16.594717 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:16.596962 (+ 2245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:16.596967 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:16.599681 (+ 2714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:16.599698 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:16.601643 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:16.601649 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:16.601891 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:16.601955 (+ 64us) client_negotiation.cc:769] Sending connection context
0928 15:01:16.602089 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 15:01:16.602285 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":737,"mutex_wait_us":80,"thread_start_us":588,"threads_started":1}
I20260928 15:01:16.603273 17698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:16.592722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44340 (local address 127.31.40.2:38827)
0928 15:01:16.593886 (+ 1164us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:16.593894 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:16.594270 (+ 376us) server_negotiation.cc:407] Connection header received
0928 15:01:16.594378 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:16.594387 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:16.594632 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:16.594838 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:16.595698 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:16.597159 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:16.599951 (+ 2792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:16.601423 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:16.601557 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:16.601848 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:16.602731 (+ 883us) server_negotiation.cc:299] Negotiation successful
0928 15:01:16.602929 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1005,"thread_start_us":885,"threads_started":1}
I20260928 15:01:16.603690 17696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:16.592205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.2:38827 (local address 127.0.0.1:44340)
0928 15:01:16.593983 (+ 1778us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:16.594001 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:16.594129 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:16.595074 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:16.595079 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:16.595104 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:16.595508 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:16.595519 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:16.597328 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:16.597333 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:16.599730 (+ 2397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:16.599741 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:16.601991 (+ 2250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:16.601997 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:16.602168 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:16.602196 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 15:01:16.602285 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 15:01:16.602399 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1647,"mutex_wait_us":69,"thread_start_us":1578,"threads_started":1}
I20260928 15:01:16.604668 17697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:16.592199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57840 (local address 127.31.40.1:39763)
0928 15:01:16.593555 (+ 1356us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:16.593565 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:16.593593 (+ 28us) server_negotiation.cc:407] Connection header received
0928 15:01:16.593703 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:16.593712 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:16.593986 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:16.594925 (+ 939us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:16.595128 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:16.596804 (+ 1676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:16.599951 (+ 3147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:16.601422 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:16.603703 (+ 2281us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:16.604064 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:16.604147 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 15:01:16.604319 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1196,"thread_start_us":1074,"threads_started":1}
I20260928 15:01:16.608716 17357 tablet_service.cc:1467] Processing CreateTablet for tablet 98469cc0c3e54dfcac895e6417f14829 (DEFAULT_TABLE table=test-table [id=d55ae185d4334f8b9b1dd00736f1bc7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:01:16.608727 17488 tablet_service.cc:1467] Processing CreateTablet for tablet 98469cc0c3e54dfcac895e6417f14829 (DEFAULT_TABLE table=test-table [id=d55ae185d4334f8b9b1dd00736f1bc7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:01:16.610136 17357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98469cc0c3e54dfcac895e6417f14829. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:16.610146 17488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98469cc0c3e54dfcac895e6417f14829. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:16.611789 17695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:16.594151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.3:36395 (local address 127.0.0.1:42618)
0928 15:01:16.603387 (+ 9236us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:16.603410 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:16.603519 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:16.605156 (+ 1637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:16.605163 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:16.605192 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:16.605373 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:16.605384 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:16.607360 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:16.607365 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:16.609219 (+ 1854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:16.609245 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:16.611035 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:16.611044 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:16.611252 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:16.611295 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 15:01:16.611367 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 15:01:16.611508 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9120,"thread_start_us":10390,"threads_started":1}
I20260928 15:01:16.611724 17422 heartbeater.cc:503] Master 127.31.40.62:42745 was elected leader, sending a full tablet report...
I20260928 15:01:16.612918 17700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:16.594524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42618 (local address 127.31.40.3:36395)
0928 15:01:16.604455 (+ 9931us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:16.604471 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:16.604507 (+ 36us) server_negotiation.cc:407] Connection header received
0928 15:01:16.604616 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:16.604626 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:16.604907 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:16.605175 (+ 268us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:16.605600 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:16.607172 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:16.609445 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:16.610848 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:16.612043 (+ 1195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:16.612328 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:16.612402 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 15:01:16.612566 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9722,"thread_start_us":9553,"threads_started":1}
I20260928 15:01:16.616883 17618 tablet_service.cc:1467] Processing CreateTablet for tablet 98469cc0c3e54dfcac895e6417f14829 (DEFAULT_TABLE table=test-table [id=d55ae185d4334f8b9b1dd00736f1bc7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:01:16.618418 17618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98469cc0c3e54dfcac895e6417f14829. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:16.623538 17701 tablet_bootstrap.cc:492] T 98469cc0c3e54dfcac895e6417f14829 P e7f64b778c544179b8a81c5ab424a47d: Bootstrap starting.
I20260928 15:01:16.627084 17701 tablet_bootstrap.cc:654] T 98469cc0c3e54dfcac895e6417f14829 P e7f64b778c544179b8a81c5ab424a47d: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:16.628098 17702 tablet_bootstrap.cc:492] T 98469cc0c3e54dfcac895e6417f14829 P 43aa27a441984beab35828e1abfd3eeb: Bootstrap starting.
I20260928 15:01:16.628657 17701 log.cc:824] T 98469cc0c3e54dfcac895e6417f14829 P e7f64b778c544179b8a81c5ab424a47d: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:16.631666 17702 tablet_bootstrap.cc:654] T 98469cc0c3e54dfcac895e6417f14829 P 43aa27a441984beab35828e1abfd3eeb: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:16.631996 17701 tablet_bootstrap.cc:492] T 98469cc0c3e54dfcac895e6417f14829 P e7f64b778c544179b8a81c5ab424a47d: No bootstrap required, opened a new log
I20260928 15:01:16.632323 17701 ts_tablet_manager.cc:1397] T 98469cc0c3e54dfcac895e6417f14829 P e7f64b778c544179b8a81c5ab424a47d: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260928 15:01:16.633010 17702 log.cc:824] T 98469cc0c3e54dfcac895e6417f14829 P 43aa27a441984beab35828e1abfd3eeb: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:16.633256 17704 tablet_bootstrap.cc:492] T 98469cc0c3e54dfcac895e6417f14829 P 4f2ab89e862240cb9295ec34283cca12: Bootstrap starting.
I20260928 15:01:16.636636 17704 tablet_bootstrap.cc:654] T 98469cc0c3e54dfcac895e6417f14829 P 4f2ab89e862240cb9295ec34283cca12: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:16.638098 17704 log.cc:824] T 98469cc0c3e54dfcac895e6417f14829 P 4f2ab89e862240cb9295ec34283cca12: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:16.638711 17702 tablet_bootstrap.cc:492] T 98469cc0c3e54dfcac895e6417f14829 P 43aa27a441984beab35828e1abfd3eeb: No bootstrap required, opened a new log
I20260928 15:01:16.639062 17702 ts_tablet_manager.cc:1397] T 98469cc0c3e54dfcac895e6417f14829 P 43aa27a441984beab35828e1abfd3eeb: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20260928 15:01:16.640980 17704 tablet_bootstrap.cc:492] T 98469cc0c3e54dfcac895e6417f14829 P 4f2ab89e862240cb9295ec34283cca12: No bootstrap required, opened a new log
I20260928 15:01:16.641253 17704 ts_tablet_manager.cc:1397] T 98469cc0c3e54dfcac895e6417f14829 P 4f2ab89e862240cb9295ec34283cca12: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260928 15:01:16.643679 17701 raft_consensus.cc:348] T 98469cc0c3e54dfcac895e6417f14829 P e7f64b778c544179b8a81c5ab424a47d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43aa27a441984beab35828e1abfd3eeb" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 39763 } } peers { permanent_uuid: "e7f64b778c544179b8a81c5ab424a47d" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 38827 } } peers { permanent_uuid: "4f2ab89e862240cb9295ec34283cca12" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 36395 } }
I20260928 15:01:16.644192 17701 raft_consensus.cc:374] T 98469cc0c3e54dfcac895e6417f14829 P e7f64b778c544179b8a81c5ab424a47d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:16.644306 17701 raft_consensus.cc:729] T 98469cc0c3e54dfcac895e6417f14829 P e7f64b778c544179b8a81c5ab424a47d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7f64b778c544179b8a81c5ab424a47d, State: Initialized, Role: FOLLOWER
I20260928 15:01:16.645023 17701 consensus_queue.cc:260] T 98469cc0c3e54dfcac895e6417f14829 P e7f64b778c544179b8a81c5ab424a47d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43aa27a441984beab35828e1abfd3eeb" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 39763 } } peers { permanent_uuid: "e7f64b778c544179b8a81c5ab424a47d" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 38827 } } peers { permanent_uuid: "4f2ab89e862240cb9295ec34283cca12" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 36395 } }
I20260928 15:01:16.648746 17553 heartbeater.cc:503] Master 127.31.40.62:42745 was elected leader, sending a full tablet report...
I20260928 15:01:16.649672 17701 ts_tablet_manager.cc:1428] T 98469cc0c3e54dfcac895e6417f14829 P e7f64b778c544179b8a81c5ab424a47d: Time spent starting tablet: real 0.017s user 0.009s sys 0.007s
I20260928 15:01:16.650725 17704 raft_consensus.cc:348] T 98469cc0c3e54dfcac895e6417f14829 P 4f2ab89e862240cb9295ec34283cca12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43aa27a441984beab35828e1abfd3eeb" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 39763 } } peers { permanent_uuid: "e7f64b778c544179b8a81c5ab424a47d" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 38827 } } peers { permanent_uuid: "4f2ab89e862240cb9295ec34283cca12" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 36395 } }
I20260928 15:01:16.651214 17704 raft_consensus.cc:374] T 98469cc0c3e54dfcac895e6417f14829 P 4f2ab89e862240cb9295ec34283cca12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:16.651335 17704 raft_consensus.cc:729] T 98469cc0c3e54dfcac895e6417f14829 P 4f2ab89e862240cb9295ec34283cca12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f2ab89e862240cb9295ec34283cca12, State: Initialized, Role: FOLLOWER
I20260928 15:01:16.651377 17702 raft_consensus.cc:348] T 98469cc0c3e54dfcac895e6417f14829 P 43aa27a441984beab35828e1abfd3eeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43aa27a441984beab35828e1abfd3eeb" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 39763 } } peers { permanent_uuid: "e7f64b778c544179b8a81c5ab424a47d" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 38827 } } peers { permanent_uuid: "4f2ab89e862240cb9295ec34283cca12" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 36395 } }
I20260928 15:01:16.651944 17702 raft_consensus.cc:374] T 98469cc0c3e54dfcac895e6417f14829 P 43aa27a441984beab35828e1abfd3eeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:16.652061 17702 raft_consensus.cc:729] T 98469cc0c3e54dfcac895e6417f14829 P 43aa27a441984beab35828e1abfd3eeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43aa27a441984beab35828e1abfd3eeb, State: Initialized, Role: FOLLOWER
I20260928 15:01:16.652103 17704 consensus_queue.cc:260] T 98469cc0c3e54dfcac895e6417f14829 P 4f2ab89e862240cb9295ec34283cca12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43aa27a441984beab35828e1abfd3eeb" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 39763 } } peers { permanent_uuid: "e7f64b778c544179b8a81c5ab424a47d" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 38827 } } peers { permanent_uuid: "4f2ab89e862240cb9295ec34283cca12" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 36395 } }
I20260928 15:01:16.652742 17702 consensus_queue.cc:260] T 98469cc0c3e54dfcac895e6417f14829 P 43aa27a441984beab35828e1abfd3eeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43aa27a441984beab35828e1abfd3eeb" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 39763 } } peers { permanent_uuid: "e7f64b778c544179b8a81c5ab424a47d" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 38827 } } peers { permanent_uuid: "4f2ab89e862240cb9295ec34283cca12" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 36395 } }
I20260928 15:01:16.653441 17683 heartbeater.cc:503] Master 127.31.40.62:42745 was elected leader, sending a full tablet report...
I20260928 15:01:16.654166 17704 ts_tablet_manager.cc:1428] T 98469cc0c3e54dfcac895e6417f14829 P 4f2ab89e862240cb9295ec34283cca12: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260928 15:01:16.658438 17702 ts_tablet_manager.cc:1428] T 98469cc0c3e54dfcac895e6417f14829 P 43aa27a441984beab35828e1abfd3eeb: Time spent starting tablet: real 0.019s user 0.014s sys 0.004s
W20260928 15:01:16.783361 17684 tablet.cc:2369] T 98469cc0c3e54dfcac895e6417f14829 P 4f2ab89e862240cb9295ec34283cca12: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:01:16.791673 17708 raft_consensus.cc:482] T 98469cc0c3e54dfcac895e6417f14829 P 4f2ab89e862240cb9295ec34283cca12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:01:16.791997 17708 raft_consensus.cc:504] T 98469cc0c3e54dfcac895e6417f14829 P 4f2ab89e862240cb9295ec34283cca12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43aa27a441984beab35828e1abfd3eeb" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 39763 } } peers { permanent_uuid: "e7f64b778c544179b8a81c5ab424a47d" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 38827 } } peers { permanent_uuid: "4f2ab89e862240cb9295ec34283cca12" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 36395 } }
I20260928 15:01:16.793665 17708 leader_election.cc:288] T 98469cc0c3e54dfcac895e6417f14829 P 4f2ab89e862240cb9295ec34283cca12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43aa27a441984beab35828e1abfd3eeb (127.31.40.1:39763), e7f64b778c544179b8a81c5ab424a47d (127.31.40.2:38827)
I20260928 15:01:16.800573 17697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:16.794280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:58467 (local address 127.31.40.1:39763)
0928 15:01:16.794521 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:16.794529 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:16.794700 (+ 171us) server_negotiation.cc:407] Connection header received
0928 15:01:16.794944 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:16.794949 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:16.795021 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:16.795105 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:16.795620 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:16.796575 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:16.798518 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:16.799597 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:16.799691 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:16.799761 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:16.800241 (+ 480us) server_negotiation.cc:299] Negotiation successful
0928 15:01:16.800328 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260928 15:01:16.800700 17685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:16.794181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:39763 (local address 127.31.40.3:58467)
0928 15:01:16.794576 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:16.794599 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:16.794766 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:16.795164 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:16.795170 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:16.795192 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:16.795428 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:16.795461 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:16.796723 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:16.796734 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:16.798357 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:16.798369 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:16.799851 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:16.799858 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:16.800071 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:16.800107 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 15:01:16.800218 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 15:01:16.800370 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268}
I20260928 15:01:16.801395 17710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:16.794277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.2:38827 (local address 127.31.40.3:42889)
0928 15:01:16.795041 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:16.795053 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:16.795185 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:16.795517 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:16.795524 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:16.795554 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:16.796028 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:16.796043 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:16.797516 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:16.797519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:16.799100 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:16.799115 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:16.800566 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:16.800573 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:16.800771 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:16.800807 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 15:01:16.800960 (+ 153us) client_negotiation.cc:240] Negotiation successful
0928 15:01:16.801100 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"thread_start_us":418,"threads_started":1}
I20260928 15:01:16.801426 17698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:16.794180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:42889 (local address 127.31.40.2:38827)
0928 15:01:16.794406 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:16.794412 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:16.795171 (+ 759us) server_negotiation.cc:407] Connection header received
0928 15:01:16.795316 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:16.795320 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:16.795391 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:16.795462 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:16.796212 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:16.797365 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:16.799312 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:16.800394 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:16.800520 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:16.800610 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:16.801033 (+ 423us) server_negotiation.cc:299] Negotiation successful
0928 15:01:16.801173 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 15:01:16.801970 17377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98469cc0c3e54dfcac895e6417f14829" candidate_uuid: "4f2ab89e862240cb9295ec34283cca12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43aa27a441984beab35828e1abfd3eeb" is_pre_election: true
I20260928 15:01:16.802291 17508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98469cc0c3e54dfcac895e6417f14829" candidate_uuid: "4f2ab89e862240cb9295ec34283cca12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7f64b778c544179b8a81c5ab424a47d" is_pre_election: true
I20260928 15:01:16.802498 17377 raft_consensus.cc:2454] T 98469cc0c3e54dfcac895e6417f14829 P 43aa27a441984beab35828e1abfd3eeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f2ab89e862240cb9295ec34283cca12 in term 0.
I20260928 15:01:16.802800 17508 raft_consensus.cc:2454] T 98469cc0c3e54dfcac895e6417f14829 P e7f64b778c544179b8a81c5ab424a47d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f2ab89e862240cb9295ec34283cca12 in term 0.
I20260928 15:01:16.803306 17572 leader_election.cc:302] T 98469cc0c3e54dfcac895e6417f14829 P 4f2ab89e862240cb9295ec34283cca12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43aa27a441984beab35828e1abfd3eeb, 4f2ab89e862240cb9295ec34283cca12; no voters:
I20260928 15:01:16.803697 17708 raft_consensus.cc:2781] T 98469cc0c3e54dfcac895e6417f14829 P 4f2ab89e862240cb9295ec34283cca12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:01:16.803853 17708 raft_consensus.cc:482] T 98469cc0c3e54dfcac895e6417f14829 P 4f2ab89e862240cb9295ec34283cca12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:01:16.804052 17708 raft_consensus.cc:3037] T 98469cc0c3e54dfcac895e6417f14829 P 4f2ab89e862240cb9295ec34283cca12 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:16.806195 17708 raft_consensus.cc:504] T 98469cc0c3e54dfcac895e6417f14829 P 4f2ab89e862240cb9295ec34283cca12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43aa27a441984beab35828e1abfd3eeb" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 39763 } } peers { permanent_uuid: "e7f64b778c544179b8a81c5ab424a47d" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 38827 } } peers { permanent_uuid: "4f2ab89e862240cb9295ec34283cca12" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 36395 } }
I20260928 15:01:16.806805 17708 leader_election.cc:288] T 98469cc0c3e54dfcac895e6417f14829 P 4f2ab89e862240cb9295ec34283cca12 [CANDIDATE]: Term 1 election: Requested vote from peers 43aa27a441984beab35828e1abfd3eeb (127.31.40.1:39763), e7f64b778c544179b8a81c5ab424a47d (127.31.40.2:38827)
I20260928 15:01:16.807291 17508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98469cc0c3e54dfcac895e6417f14829" candidate_uuid: "4f2ab89e862240cb9295ec34283cca12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7f64b778c544179b8a81c5ab424a47d"
I20260928 15:01:16.807307 17377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98469cc0c3e54dfcac895e6417f14829" candidate_uuid: "4f2ab89e862240cb9295ec34283cca12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43aa27a441984beab35828e1abfd3eeb"
I20260928 15:01:16.807492 17508 raft_consensus.cc:3037] T 98469cc0c3e54dfcac895e6417f14829 P e7f64b778c544179b8a81c5ab424a47d [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:16.807513 17377 raft_consensus.cc:3037] T 98469cc0c3e54dfcac895e6417f14829 P 43aa27a441984beab35828e1abfd3eeb [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:16.809903 17377 raft_consensus.cc:2454] T 98469cc0c3e54dfcac895e6417f14829 P 43aa27a441984beab35828e1abfd3eeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f2ab89e862240cb9295ec34283cca12 in term 1.
I20260928 15:01:16.810053 17508 raft_consensus.cc:2454] T 98469cc0c3e54dfcac895e6417f14829 P e7f64b778c544179b8a81c5ab424a47d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f2ab89e862240cb9295ec34283cca12 in term 1.
I20260928 15:01:16.810334 17572 leader_election.cc:302] T 98469cc0c3e54dfcac895e6417f14829 P 4f2ab89e862240cb9295ec34283cca12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43aa27a441984beab35828e1abfd3eeb, 4f2ab89e862240cb9295ec34283cca12; no voters:
I20260928 15:01:16.810614 17708 raft_consensus.cc:2781] T 98469cc0c3e54dfcac895e6417f14829 P 4f2ab89e862240cb9295ec34283cca12 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:01:16.811044 17708 raft_consensus.cc:686] T 98469cc0c3e54dfcac895e6417f14829 P 4f2ab89e862240cb9295ec34283cca12 [term 1 LEADER]: Becoming Leader. State: Replica: 4f2ab89e862240cb9295ec34283cca12, State: Running, Role: LEADER
I20260928 15:01:16.811512 17708 consensus_queue.cc:237] T 98469cc0c3e54dfcac895e6417f14829 P 4f2ab89e862240cb9295ec34283cca12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43aa27a441984beab35828e1abfd3eeb" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 39763 } } peers { permanent_uuid: "e7f64b778c544179b8a81c5ab424a47d" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 38827 } } peers { permanent_uuid: "4f2ab89e862240cb9295ec34283cca12" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 36395 } }
I20260928 15:01:16.816083 17235 catalog_manager.cc:5495] T 98469cc0c3e54dfcac895e6417f14829 P 4f2ab89e862240cb9295ec34283cca12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f2ab89e862240cb9295ec34283cca12 (127.31.40.3). New cstate: current_term: 1 leader_uuid: "4f2ab89e862240cb9295ec34283cca12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43aa27a441984beab35828e1abfd3eeb" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 39763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7f64b778c544179b8a81c5ab424a47d" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 38827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f2ab89e862240cb9295ec34283cca12" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 36395 } health_report { overall_health: HEALTHY } } }
W20260928 15:01:16.817411 17554 tablet.cc:2369] T 98469cc0c3e54dfcac895e6417f14829 P e7f64b778c544179b8a81c5ab424a47d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:01:16.840093 17423 tablet.cc:2369] T 98469cc0c3e54dfcac895e6417f14829 P 43aa27a441984beab35828e1abfd3eeb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:01:16.855896 17700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:16.850010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42624 (local address 127.31.40.3:36395)
0928 15:01:16.850257 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:16.850261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:16.850280 (+ 19us) server_negotiation.cc:407] Connection header received
0928 15:01:16.850407 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:16.850411 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:16.850491 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:16.850616 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:01:16.850945 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:16.851652 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:16.853243 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:16.853695 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:16.853888 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:16.854678 (+ 790us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:01:16.855214 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:01:16.855343 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:16.855437 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 15:01:16.855504 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 15:01:16.864953 17377 raft_consensus.cc:1261] T 98469cc0c3e54dfcac895e6417f14829 P 43aa27a441984beab35828e1abfd3eeb [term 1 FOLLOWER]: Refusing update from remote peer 4f2ab89e862240cb9295ec34283cca12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:01:16.864944 17508 raft_consensus.cc:1261] T 98469cc0c3e54dfcac895e6417f14829 P e7f64b778c544179b8a81c5ab424a47d [term 1 FOLLOWER]: Refusing update from remote peer 4f2ab89e862240cb9295ec34283cca12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:01:16.865942 17711 consensus_queue.cc:1035] T 98469cc0c3e54dfcac895e6417f14829 P 4f2ab89e862240cb9295ec34283cca12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43aa27a441984beab35828e1abfd3eeb" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 39763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:01:16.866323 17708 consensus_queue.cc:1035] T 98469cc0c3e54dfcac895e6417f14829 P 4f2ab89e862240cb9295ec34283cca12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7f64b778c544179b8a81c5ab424a47d" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 38827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:01:16.878340 17715 mvcc.cc:205] Tried to move back new op lower bound from 7334329044427579392 to 7334329044223942656. Current Snapshot: MvccSnapshot[applied={T|T < 7334329044427579392}]
I20260928 15:01:16.889369 17716 mvcc.cc:205] Tried to move back new op lower bound from 7334329044427579392 to 7334329044223942656. Current Snapshot: MvccSnapshot[applied={T|T < 7334329044427579392}]
I20260928 15:01:16.891549 17234 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:01:16.891913 17234 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:01:16.895656 17234 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6720069b68af4fcfbc46d12a737cfe62: Sending DeleteTablet for 3 replicas of tablet 98469cc0c3e54dfcac895e6417f14829
I20260928 15:01:16.896924 17357 tablet_service.cc:1514] Processing DeleteTablet for tablet 98469cc0c3e54dfcac895e6417f14829 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:01:16 UTC) from {username='slave'} at 127.0.0.1:57840
I20260928 15:01:16.896947 17618 tablet_service.cc:1514] Processing DeleteTablet for tablet 98469cc0c3e54dfcac895e6417f14829 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:01:16 UTC) from {username='slave'} at 127.0.0.1:42618
I20260928 15:01:16.896941 17488 tablet_service.cc:1514] Processing DeleteTablet for tablet 98469cc0c3e54dfcac895e6417f14829 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:01:16 UTC) from {username='slave'} at 127.0.0.1:44340
I20260928 15:01:16.898977 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 17295
I20260928 15:01:16.912364 17725 tablet_replica.cc:330] T 98469cc0c3e54dfcac895e6417f14829 P 4f2ab89e862240cb9295ec34283cca12: stopping tablet replica
I20260928 15:01:16.912828 17725 raft_consensus.cc:2229] T 98469cc0c3e54dfcac895e6417f14829 P 4f2ab89e862240cb9295ec34283cca12 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:01:16.913443 17725 raft_consensus.cc:2258] T 98469cc0c3e54dfcac895e6417f14829 P 4f2ab89e862240cb9295ec34283cca12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:01:16.914268 17726 tablet_replica.cc:330] T 98469cc0c3e54dfcac895e6417f14829 P e7f64b778c544179b8a81c5ab424a47d: stopping tablet replica
I20260928 15:01:16.914855 17726 raft_consensus.cc:2229] T 98469cc0c3e54dfcac895e6417f14829 P e7f64b778c544179b8a81c5ab424a47d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:01:16.915452 17726 raft_consensus.cc:2258] T 98469cc0c3e54dfcac895e6417f14829 P e7f64b778c544179b8a81c5ab424a47d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:01:16.916004 17725 ts_tablet_manager.cc:1905] T 98469cc0c3e54dfcac895e6417f14829 P 4f2ab89e862240cb9295ec34283cca12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:01:16.917717 17726 ts_tablet_manager.cc:1905] T 98469cc0c3e54dfcac895e6417f14829 P e7f64b778c544179b8a81c5ab424a47d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:01:16.923460 17725 ts_tablet_manager.cc:1918] T 98469cc0c3e54dfcac895e6417f14829 P 4f2ab89e862240cb9295ec34283cca12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:01:16.923678 17725 log.cc:1196] T 98469cc0c3e54dfcac895e6417f14829 P 4f2ab89e862240cb9295ec34283cca12: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/ts-2/wal/wals/98469cc0c3e54dfcac895e6417f14829
I20260928 15:01:16.924302 17725 ts_tablet_manager.cc:1939] T 98469cc0c3e54dfcac895e6417f14829 P 4f2ab89e862240cb9295ec34283cca12: Deleting consensus metadata
I20260928 15:01:16.925015 17726 ts_tablet_manager.cc:1918] T 98469cc0c3e54dfcac895e6417f14829 P e7f64b778c544179b8a81c5ab424a47d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:01:16.925251 17726 log.cc:1196] T 98469cc0c3e54dfcac895e6417f14829 P e7f64b778c544179b8a81c5ab424a47d: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/ts-1/wal/wals/98469cc0c3e54dfcac895e6417f14829
I20260928 15:01:16.925863 17726 ts_tablet_manager.cc:1939] T 98469cc0c3e54dfcac895e6417f14829 P e7f64b778c544179b8a81c5ab424a47d: Deleting consensus metadata
I20260928 15:01:16.927397 17223 catalog_manager.cc:4841] TS e7f64b778c544179b8a81c5ab424a47d (127.31.40.2:38827): tablet 98469cc0c3e54dfcac895e6417f14829 (table test-table [id=d55ae185d4334f8b9b1dd00736f1bc7d]) successfully deleted
I20260928 15:01:16.927739 17223 catalog_manager.cc:4841] TS 4f2ab89e862240cb9295ec34283cca12 (127.31.40.3:36395): tablet 98469cc0c3e54dfcac895e6417f14829 (table test-table [id=d55ae185d4334f8b9b1dd00736f1bc7d]) successfully deleted
I20260928 15:01:16.928310 17724 tablet_replica.cc:330] T 98469cc0c3e54dfcac895e6417f14829 P 43aa27a441984beab35828e1abfd3eeb: stopping tablet replica
I20260928 15:01:16.928704 17724 raft_consensus.cc:2229] T 98469cc0c3e54dfcac895e6417f14829 P 43aa27a441984beab35828e1abfd3eeb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:01:16.929185 17724 raft_consensus.cc:2258] T 98469cc0c3e54dfcac895e6417f14829 P 43aa27a441984beab35828e1abfd3eeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:01:16.931358 17724 ts_tablet_manager.cc:1905] T 98469cc0c3e54dfcac895e6417f14829 P 43aa27a441984beab35828e1abfd3eeb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:01:16.938298 17724 ts_tablet_manager.cc:1918] T 98469cc0c3e54dfcac895e6417f14829 P 43aa27a441984beab35828e1abfd3eeb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:01:16.938501 17724 log.cc:1196] T 98469cc0c3e54dfcac895e6417f14829 P 43aa27a441984beab35828e1abfd3eeb: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607365362981-31904-0/minicluster-data/ts-0/wal/wals/98469cc0c3e54dfcac895e6417f14829
I20260928 15:01:16.939050 17724 ts_tablet_manager.cc:1939] T 98469cc0c3e54dfcac895e6417f14829 P 43aa27a441984beab35828e1abfd3eeb: Deleting consensus metadata
I20260928 15:01:16.940068 17697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:16.900114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57842 (local address 127.31.40.1:39763)
0928 15:01:16.900302 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:16.900310 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:16.924468 (+ 24158us) server_negotiation.cc:407] Connection header received
0928 15:01:16.924996 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:16.925003 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:16.925143 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:16.925225 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:16.925933 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:16.927082 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:16.929308 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:16.930050 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:16.930224 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:16.938429 (+ 8205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:16.938443 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:16.938463 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:01:16.939205 (+ 742us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:16.939262 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:01:16.939284 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:16.939419 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:16.939528 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 15:01:16.939645 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 15:01:16.940765 17417 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:01:16.941112 17221 catalog_manager.cc:4841] TS 43aa27a441984beab35828e1abfd3eeb (127.31.40.1:39763): tablet 98469cc0c3e54dfcac895e6417f14829 (table test-table [id=d55ae185d4334f8b9b1dd00736f1bc7d]) successfully deleted
I20260928 15:01:17.033056 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 17295
I20260928 15:01:17.052268 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 17426
I20260928 15:01:17.059463 17698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:17.053199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44346 (local address 127.31.40.2:38827)
0928 15:01:17.053428 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:17.053434 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:17.053457 (+ 23us) server_negotiation.cc:407] Connection header received
0928 15:01:17.053535 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:17.053540 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:17.053659 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:17.053775 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:17.054194 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:17.054981 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:17.056336 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:17.056843 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:17.056982 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:17.058057 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:17.058067 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:17.058082 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:01:17.058686 (+ 604us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:17.058736 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:01:17.058755 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:17.058898 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:17.059031 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 15:01:17.059141 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 15:01:17.060359 17548 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:01:17.170065 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 17426
I20260928 15:01:17.189373 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 17556
I20260928 15:01:17.196703 17700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:17.190179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42626 (local address 127.31.40.3:36395)
0928 15:01:17.190378 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:17.190386 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:17.190409 (+ 23us) server_negotiation.cc:407] Connection header received
0928 15:01:17.190626 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:17.190631 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:17.190728 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:17.190841 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:17.191257 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:17.192143 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:17.193468 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:17.193923 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:17.194056 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:17.195218 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:17.195227 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:17.195245 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:01:17.195856 (+ 611us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:17.195968 (+ 112us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:01:17.195987 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:17.196095 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:17.196243 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 15:01:17.196343 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 15:01:17.197373 17678 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:01:17.321877 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 17556
I20260928 15:01:17.342916 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 17205
I20260928 15:01:17.344158 17266 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:01:17.454253 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 17205
2026-09-28T15:01:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2952 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:01:17 dist-test-slave-j968 krb5kdc[17734](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:01:17 dist-test-slave-j968 krb5kdc[17734](info): set up 2 sockets
Sep 28 15:01:17 dist-test-slave-j968 krb5kdc[17734](info): commencing operation
krb5kdc: starting...
W20260928 15:01:18.577193 31904 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Sep 28 15:01:18 dist-test-slave-j968 krb5kdc[17734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:01:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:01:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab.
Entry for principal kudu/127.31.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab.
I20260928 15:01:18.836098 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:39741
--webserver_interface=127.31.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:36523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.40.62:39741
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:01:19.169727 17747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:19.170042 17747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:19.170209 17747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:19.177568 17747 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:01:19.177680 17747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:19.177778 17747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:19.177892 17747 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:01:19.177975 17747 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:01:19.187930 17747 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:36523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.40.62:39741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:39741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:01:19.189347 17747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:19.190696 17747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:19.202834 17752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:19.202833 17755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:19.202877 17753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:19.203680 17747 server_base.cc:1034] running on GCE node
I20260928 15:01:19.204469 17747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:19.205897 17747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:19.207096 17747 hybrid_clock.cc:648] HybridClock initialized: now 1790607679207046 us; error 42 us; skew 500 ppm
I20260928 15:01:19.207484 17747 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:01:19 dist-test-slave-j968 krb5kdc[17734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607679, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:01:19.210600 17747 init.cc:377] Logged in from keytab as kudu/127.31.40.62@KRBTEST.COM (short username kudu)
I20260928 15:01:19.212169 17747 webserver.cc:460] Webserver started at http://127.31.40.62:44571/ using document root <none> and password file <none>
I20260928 15:01:19.212809 17747 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:19.212915 17747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:19.213163 17747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:19.216254 17747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/master-0/data/instance:
uuid: "37d0a56e7cba49a8b0bfdc6af53f031c"
format_stamp: "Formatted at 2026-09-28 15:01:19 on dist-test-slave-j968"
server_key: "399b9c0204ed52984047ca788a48c7ab"
server_key_iv: "03018ae2f5d4f4dc98277aa4582e4cc8"
server_key_version: "encryptionkey@0"
I20260928 15:01:19.216993 17747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/master-0/wal/instance:
uuid: "37d0a56e7cba49a8b0bfdc6af53f031c"
format_stamp: "Formatted at 2026-09-28 15:01:19 on dist-test-slave-j968"
server_key: "399b9c0204ed52984047ca788a48c7ab"
server_key_iv: "03018ae2f5d4f4dc98277aa4582e4cc8"
server_key_version: "encryptionkey@0"
I20260928 15:01:19.222316 17747 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 15:01:19.226046 17762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:19.227451 17747 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 15:01:19.227640 17747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/master-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/master-0/wal
uuid: "37d0a56e7cba49a8b0bfdc6af53f031c"
format_stamp: "Formatted at 2026-09-28 15:01:19 on dist-test-slave-j968"
server_key: "399b9c0204ed52984047ca788a48c7ab"
server_key_iv: "03018ae2f5d4f4dc98277aa4582e4cc8"
server_key_version: "encryptionkey@0"
I20260928 15:01:19.227857 17747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:19.254985 17747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:19.259238 17747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:19.259537 17747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:19.278599 17747 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.62:39741
I20260928 15:01:19.278630 17813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.62:39741 every 8 connection(s)
I20260928 15:01:19.280040 17747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
I20260928 15:01:19.281414 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 17747
I20260928 15:01:19.281605 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/master-0/wal/instance
I20260928 15:01:19.282240 31904 external_mini_cluster.cc:1428] Setting key 13b1b6282ec778b26a6de052a062ed81
I20260928 15:01:19.285064 17814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 15:01:19 dist-test-slave-j968 krb5kdc[17734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 15:01:19.298724 17814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37d0a56e7cba49a8b0bfdc6af53f031c: Bootstrap starting.
I20260928 15:01:19.302348 17814 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37d0a56e7cba49a8b0bfdc6af53f031c: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:19.303417 17814 log.cc:824] T 00000000000000000000000000000000 P 37d0a56e7cba49a8b0bfdc6af53f031c: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:19.306244 17817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:19.284247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36048 (local address 127.31.40.62:39741)
0928 15:01:19.285025 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:19.285037 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:19.285081 (+ 44us) server_negotiation.cc:407] Connection header received
0928 15:01:19.286764 (+ 1683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:19.286786 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:19.287158 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:19.287494 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:19.289475 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:19.290982 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:19.292891 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:19.293547 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:19.293673 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:19.298810 (+ 5137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:19.298818 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:19.298834 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:19.298908 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:19.302023 (+ 3115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:19.302843 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:19.302850 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:19.302857 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:19.303030 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:19.303433 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:19.303437 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:19.303439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:19.304058 (+ 619us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:19.304407 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:19.304841 (+ 434us) server_negotiation.cc:299] Negotiation successful
0928 15:01:19.305279 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":350,"threads_started":1}
I20260928 15:01:19.306530 17814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37d0a56e7cba49a8b0bfdc6af53f031c: No bootstrap required, opened a new log
I20260928 15:01:19.314544 17814 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37d0a56e7cba49a8b0bfdc6af53f031c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37d0a56e7cba49a8b0bfdc6af53f031c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 39741 } }
I20260928 15:01:19.314977 17814 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37d0a56e7cba49a8b0bfdc6af53f031c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:19.315085 17814 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37d0a56e7cba49a8b0bfdc6af53f031c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37d0a56e7cba49a8b0bfdc6af53f031c, State: Initialized, Role: FOLLOWER
I20260928 15:01:19.315640 17814 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37d0a56e7cba49a8b0bfdc6af53f031c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37d0a56e7cba49a8b0bfdc6af53f031c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 39741 } }
I20260928 15:01:19.315924 17814 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37d0a56e7cba49a8b0bfdc6af53f031c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:01:19.316108 17814 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37d0a56e7cba49a8b0bfdc6af53f031c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:01:19.316311 17814 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37d0a56e7cba49a8b0bfdc6af53f031c [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:19.318498 17814 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37d0a56e7cba49a8b0bfdc6af53f031c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37d0a56e7cba49a8b0bfdc6af53f031c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 39741 } }
I20260928 15:01:19.319006 17814 leader_election.cc:302] T 00000000000000000000000000000000 P 37d0a56e7cba49a8b0bfdc6af53f031c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37d0a56e7cba49a8b0bfdc6af53f031c; no voters:
I20260928 15:01:19.319607 17814 leader_election.cc:288] T 00000000000000000000000000000000 P 37d0a56e7cba49a8b0bfdc6af53f031c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:01:19.319787 17819 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37d0a56e7cba49a8b0bfdc6af53f031c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:01:19.321136 17819 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37d0a56e7cba49a8b0bfdc6af53f031c [term 1 LEADER]: Becoming Leader. State: Replica: 37d0a56e7cba49a8b0bfdc6af53f031c, State: Running, Role: LEADER
I20260928 15:01:19.321679 17819 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37d0a56e7cba49a8b0bfdc6af53f031c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37d0a56e7cba49a8b0bfdc6af53f031c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 39741 } }
I20260928 15:01:19.322196 17814 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37d0a56e7cba49a8b0bfdc6af53f031c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:01:19.324757 17820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37d0a56e7cba49a8b0bfdc6af53f031c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37d0a56e7cba49a8b0bfdc6af53f031c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37d0a56e7cba49a8b0bfdc6af53f031c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 39741 } } }
I20260928 15:01:19.324751 17821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37d0a56e7cba49a8b0bfdc6af53f031c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37d0a56e7cba49a8b0bfdc6af53f031c. Latest consensus state: current_term: 1 leader_uuid: "37d0a56e7cba49a8b0bfdc6af53f031c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37d0a56e7cba49a8b0bfdc6af53f031c" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 39741 } } }
I20260928 15:01:19.325197 17820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37d0a56e7cba49a8b0bfdc6af53f031c [sys.catalog]: This master's current role is: LEADER
I20260928 15:01:19.325227 17821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37d0a56e7cba49a8b0bfdc6af53f031c [sys.catalog]: This master's current role is: LEADER
I20260928 15:01:19.327605 17826 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:01:19.334375 17826 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:01:19.345546 17826 catalog_manager.cc:1347] Generated new cluster ID: 86ca331fbf2845689abcb2d0084846ed
I20260928 15:01:19.345682 17826 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:01:19.354151 17826 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:01:19.355525 17826 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:01:19.369856 17826 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37d0a56e7cba49a8b0bfdc6af53f031c: Generated new TSK 0
I20260928 15:01:19.370998 17826 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.40.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab.
Entry for principal kudu/127.31.40.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab.
I20260928 15:01:19.440719 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.1:0
--local_ip_for_outbound_sockets=127.31.40.1
--webserver_interface=127.31.40.1
--webserver_port=0
--tserver_master_addrs=127.31.40.62:39741
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab
--principal=kudu/127.31.40.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:36523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:01:19.787670 17840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:19.788092 17840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:19.788267 17840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:19.795791 17840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:19.796026 17840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:19.796200 17840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.1
I20260928 15:01:19.806654 17840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:36523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab
--principal=kudu/127.31.40.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.40.1
--webserver_port=0
--tserver_master_addrs=127.31.40.62:39741
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.1
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:01:19.808141 17840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:19.809502 17840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:19.822886 17845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:19.822909 17848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:19.822826 17846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:19.823696 17840 server_base.cc:1034] running on GCE node
I20260928 15:01:19.824369 17840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:19.825394 17840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:19.826658 17840 hybrid_clock.cc:648] HybridClock initialized: now 1790607679826609 us; error 44 us; skew 500 ppm
I20260928 15:01:19.827090 17840 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:01:19 dist-test-slave-j968 krb5kdc[17734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607679, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:01:19.830331 17840 init.cc:377] Logged in from keytab as kudu/127.31.40.1@KRBTEST.COM (short username kudu)
I20260928 15:01:19.832011 17840 webserver.cc:460] Webserver started at http://127.31.40.1:35087/ using document root <none> and password file <none>
I20260928 15:01:19.832726 17840 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:19.832871 17840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:19.833190 17840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:19.836256 17840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/ts-0/data/instance:
uuid: "dafb852b35dc45f0bafccad1c9f86ff9"
format_stamp: "Formatted at 2026-09-28 15:01:19 on dist-test-slave-j968"
server_key: "3673158d43f7a2fc64430a58a9a295d1"
server_key_iv: "d22748f1fd616d2d05e285d6e583cba5"
server_key_version: "encryptionkey@0"
I20260928 15:01:19.837054 17840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/ts-0/wal/instance:
uuid: "dafb852b35dc45f0bafccad1c9f86ff9"
format_stamp: "Formatted at 2026-09-28 15:01:19 on dist-test-slave-j968"
server_key: "3673158d43f7a2fc64430a58a9a295d1"
server_key_iv: "d22748f1fd616d2d05e285d6e583cba5"
server_key_version: "encryptionkey@0"
I20260928 15:01:19.842497 17840 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 15:01:19.846050 17855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:19.847467 17840 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 15:01:19.847645 17840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/ts-0/wal
uuid: "dafb852b35dc45f0bafccad1c9f86ff9"
format_stamp: "Formatted at 2026-09-28 15:01:19 on dist-test-slave-j968"
server_key: "3673158d43f7a2fc64430a58a9a295d1"
server_key_iv: "d22748f1fd616d2d05e285d6e583cba5"
server_key_version: "encryptionkey@0"
I20260928 15:01:19.847875 17840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:19.870011 17840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:19.874506 17840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:19.874804 17840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:19.875790 17840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:01:19.877190 17840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:01:19.877306 17840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:19.877457 17840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:01:19.877532 17840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:19.912441 17840 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.1:42919
I20260928 15:01:19.912528 17967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.1:42919 every 8 connection(s)
I20260928 15:01:19.913811 17840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
I20260928 15:01:19.920090 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 17840
I20260928 15:01:19.920324 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/ts-0/wal/instance
I20260928 15:01:19.920797 31904 external_mini_cluster.cc:1428] Setting key 1c593fa769dd88d64e6920728388bffb
Sep 28 15:01:19 dist-test-slave-j968 krb5kdc[17734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607679, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.1@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 15:01:19.936479 17970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:19.916608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:55205 (local address 127.31.40.62:39741)
0928 15:01:19.917173 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:19.917179 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:19.918069 (+ 890us) server_negotiation.cc:407] Connection header received
0928 15:01:19.920076 (+ 2007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:19.920080 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:19.920158 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:19.920288 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:19.922055 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:19.923023 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:19.924787 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:19.925206 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:19.925348 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:19.929626 (+ 4278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:19.929631 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:19.929634 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:19.929682 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:19.933010 (+ 3328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:19.933755 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:19.933764 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:19.933770 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:19.933918 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:19.934400 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:19.934408 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:19.934414 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:19.934794 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:19.934956 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:19.935691 (+ 735us) server_negotiation.cc:299] Negotiation successful
0928 15:01:19.935984 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":306,"threads_started":1}
I20260928 15:01:19.937184 17971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:19.917116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:39741 (local address 127.31.40.1:55205)
0928 15:01:19.917906 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:19.917949 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:19.919732 (+ 1783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:19.920559 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:19.920568 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:19.921140 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:01:19.921872 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:19.921891 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:19.923187 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:19.923193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:19.924586 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:19.924599 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:19.925463 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:19.925470 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:19.925709 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:19.926438 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:01:19.926461 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:01:19.929389 (+ 2928us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:01:19.933152 (+ 3763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:01:19.933159 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:01:19.933174 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:01:19.933621 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:01:19.934034 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:01:19.934037 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:01:19.934039 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:01:19.934234 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:01:19.934919 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:01:19.934925 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:01:19.935251 (+ 326us) client_negotiation.cc:769] Sending connection context
0928 15:01:19.935520 (+ 269us) client_negotiation.cc:240] Negotiation successful
0928 15:01:19.936025 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":287,"threads_started":1}
I20260928 15:01:19.938858 17968 heartbeater.cc:347] Connected to a master server at 127.31.40.62:39741
I20260928 15:01:19.939190 17968 heartbeater.cc:464] Registering TS with master...
I20260928 15:01:19.940137 17968 heartbeater.cc:511] Master 127.31.40.62:39741 requested a full tablet report, sending...
I20260928 15:01:19.942474 17779 ts_manager.cc:194] Registered new tserver with Master: dafb852b35dc45f0bafccad1c9f86ff9 (127.31.40.1:42919)
I20260928 15:01:19.944149 17779 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.1@KRBTEST.COM'} at 127.31.40.1:55205
WARNING: no policy specified for kudu/127.31.40.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab.
Entry for principal kudu/127.31.40.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab.
I20260928 15:01:19.979938 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.2:0
--local_ip_for_outbound_sockets=127.31.40.2
--webserver_interface=127.31.40.2
--webserver_port=0
--tserver_master_addrs=127.31.40.62:39741
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab
--principal=kudu/127.31.40.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:36523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:01:20.310403 17974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:20.310679 17974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:20.310833 17974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:20.318023 17974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:20.318133 17974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:20.318306 17974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.2
I20260928 15:01:20.328485 17974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:36523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab
--principal=kudu/127.31.40.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.40.2
--webserver_port=0
--tserver_master_addrs=127.31.40.62:39741
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.2
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:01:20.329794 17974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:20.331166 17974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:20.344660 17979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:20.344622 17982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:20.344687 17980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:20.345638 17974 server_base.cc:1034] running on GCE node
I20260928 15:01:20.346282 17974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:20.347280 17974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:20.348497 17974 hybrid_clock.cc:648] HybridClock initialized: now 1790607680348428 us; error 63 us; skew 500 ppm
I20260928 15:01:20.348807 17974 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:01:20 dist-test-slave-j968 krb5kdc[17734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607680, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:01:20.352069 17974 init.cc:377] Logged in from keytab as kudu/127.31.40.2@KRBTEST.COM (short username kudu)
I20260928 15:01:20.353588 17974 webserver.cc:460] Webserver started at http://127.31.40.2:42921/ using document root <none> and password file <none>
I20260928 15:01:20.354251 17974 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:20.354398 17974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:20.354702 17974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:20.357622 17974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/ts-1/data/instance:
uuid: "02b3d3f838194f669e76cc67623831fa"
format_stamp: "Formatted at 2026-09-28 15:01:20 on dist-test-slave-j968"
server_key: "05850688419bc917764b69797af57dc9"
server_key_iv: "69bf7d14388f29a8b318a1c8f59d28e9"
server_key_version: "encryptionkey@0"
I20260928 15:01:20.358393 17974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/ts-1/wal/instance:
uuid: "02b3d3f838194f669e76cc67623831fa"
format_stamp: "Formatted at 2026-09-28 15:01:20 on dist-test-slave-j968"
server_key: "05850688419bc917764b69797af57dc9"
server_key_iv: "69bf7d14388f29a8b318a1c8f59d28e9"
server_key_version: "encryptionkey@0"
I20260928 15:01:20.363662 17974 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 15:01:20.367128 17989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:20.368462 17974 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:01:20.368635 17974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/ts-1/wal
uuid: "02b3d3f838194f669e76cc67623831fa"
format_stamp: "Formatted at 2026-09-28 15:01:20 on dist-test-slave-j968"
server_key: "05850688419bc917764b69797af57dc9"
server_key_iv: "69bf7d14388f29a8b318a1c8f59d28e9"
server_key_version: "encryptionkey@0"
I20260928 15:01:20.368832 17974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:20.392987 17974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:20.397652 17974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:20.397954 17974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:20.398996 17974 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:01:20.400480 17974 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:01:20.400552 17974 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:20.400635 17974 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:01:20.400679 17974 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:20.434229 17974 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.2:35705
I20260928 15:01:20.434238 18101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.2:35705 every 8 connection(s)
I20260928 15:01:20.435638 17974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
I20260928 15:01:20.444536 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 17974
I20260928 15:01:20.444729 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/ts-1/wal/instance
I20260928 15:01:20.445171 31904 external_mini_cluster.cc:1428] Setting key 2faf2ca26bb1e33d5c61435350df57e3
Sep 28 15:01:20 dist-test-slave-j968 krb5kdc[17734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607680, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.2@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 15:01:20.456552 18104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:20.438395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:51287 (local address 127.31.40.62:39741)
0928 15:01:20.438877 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:20.438882 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:20.439674 (+ 792us) server_negotiation.cc:407] Connection header received
0928 15:01:20.441637 (+ 1963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:20.441644 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:20.441753 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:20.441901 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:20.443542 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:20.444350 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:20.446211 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:20.446710 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:20.446862 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:20.450381 (+ 3519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:20.450387 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:20.450390 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:20.450423 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:20.453424 (+ 3001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:20.454209 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:20.454216 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:20.454220 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:20.454353 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:20.454795 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:20.454801 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:20.454804 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:20.455083 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:20.455259 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:20.456081 (+ 822us) server_negotiation.cc:299] Negotiation successful
0928 15:01:20.456277 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":227,"threads_started":1}
I20260928 15:01:20.458096 18105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:20.438815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:39741 (local address 127.31.40.2:51287)
0928 15:01:20.439514 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:20.439552 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:20.441341 (+ 1789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:20.442152 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:20.442160 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:20.442784 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:01:20.443366 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:20.443384 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:20.444513 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:20.444517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:20.446053 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:20.446065 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:20.446954 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:20.446962 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:20.447206 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:20.447942 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:01:20.447966 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:01:20.450143 (+ 2177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:01:20.453580 (+ 3437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:01:20.453590 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:01:20.453607 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:01:20.454070 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:01:20.454473 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:01:20.454478 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:01:20.454482 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:01:20.454676 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:01:20.455236 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:01:20.455244 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:01:20.455661 (+ 417us) client_negotiation.cc:769] Sending connection context
0928 15:01:20.455988 (+ 327us) client_negotiation.cc:240] Negotiation successful
0928 15:01:20.456595 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":267,"threads_started":1}
I20260928 15:01:20.459661 18102 heartbeater.cc:347] Connected to a master server at 127.31.40.62:39741
I20260928 15:01:20.460016 18102 heartbeater.cc:464] Registering TS with master...
I20260928 15:01:20.460739 18102 heartbeater.cc:511] Master 127.31.40.62:39741 requested a full tablet report, sending...
I20260928 15:01:20.462248 17779 ts_manager.cc:194] Registered new tserver with Master: 02b3d3f838194f669e76cc67623831fa (127.31.40.2:35705)
I20260928 15:01:20.463097 17779 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.2@KRBTEST.COM'} at 127.31.40.2:51287
WARNING: no policy specified for kudu/127.31.40.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab.
Entry for principal kudu/127.31.40.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab.
I20260928 15:01:20.504195 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.3:0
--local_ip_for_outbound_sockets=127.31.40.3
--webserver_interface=127.31.40.3
--webserver_port=0
--tserver_master_addrs=127.31.40.62:39741
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab
--principal=kudu/127.31.40.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:36523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:01:20.840173 18108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:20.840454 18108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:20.840548 18108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:20.847836 18108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:20.847968 18108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:20.848080 18108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.3
I20260928 15:01:20.858117 18108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:36523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab
--principal=kudu/127.31.40.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.40.3
--webserver_port=0
--tserver_master_addrs=127.31.40.62:39741
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.3
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:01:20.859383 18108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:20.860673 18108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:20.873809 18113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:20.873980 18116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:20.874019 18114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:20.875604 18108 server_base.cc:1034] running on GCE node
I20260928 15:01:20.876394 18108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:20.877497 18108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:20.878722 18108 hybrid_clock.cc:648] HybridClock initialized: now 1790607680878652 us; error 58 us; skew 500 ppm
I20260928 15:01:20.879132 18108 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:01:20 dist-test-slave-j968 krb5kdc[17734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607680, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:01:20.882618 18108 init.cc:377] Logged in from keytab as kudu/127.31.40.3@KRBTEST.COM (short username kudu)
I20260928 15:01:20.884356 18108 webserver.cc:460] Webserver started at http://127.31.40.3:43565/ using document root <none> and password file <none>
I20260928 15:01:20.885123 18108 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:20.885270 18108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:20.885628 18108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:20.888929 18108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/ts-2/data/instance:
uuid: "9639a6f616c844db9095f1b2dfe9ebee"
format_stamp: "Formatted at 2026-09-28 15:01:20 on dist-test-slave-j968"
server_key: "182a6ac33126ff08d494ae277d7425af"
server_key_iv: "bb844b184dc9d4edd0ac42f8dc799921"
server_key_version: "encryptionkey@0"
I20260928 15:01:20.889789 18108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/ts-2/wal/instance:
uuid: "9639a6f616c844db9095f1b2dfe9ebee"
format_stamp: "Formatted at 2026-09-28 15:01:20 on dist-test-slave-j968"
server_key: "182a6ac33126ff08d494ae277d7425af"
server_key_iv: "bb844b184dc9d4edd0ac42f8dc799921"
server_key_version: "encryptionkey@0"
I20260928 15:01:20.903358 18108 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.006s sys 0.008s
I20260928 15:01:20.907593 18123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:20.909226 18108 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260928 15:01:20.909440 18108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/ts-2/wal
uuid: "9639a6f616c844db9095f1b2dfe9ebee"
format_stamp: "Formatted at 2026-09-28 15:01:20 on dist-test-slave-j968"
server_key: "182a6ac33126ff08d494ae277d7425af"
server_key_iv: "bb844b184dc9d4edd0ac42f8dc799921"
server_key_version: "encryptionkey@0"
I20260928 15:01:20.909648 18108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:20.932984 18108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:20.937400 18108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:20.937717 18108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:20.938824 18108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:01:20.940469 18108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:01:20.940547 18108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:20.940735 18108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:01:20.940793 18108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:20.947906 17968 heartbeater.cc:503] Master 127.31.40.62:39741 was elected leader, sending a full tablet report...
I20260928 15:01:20.977249 18108 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.3:40043
I20260928 15:01:20.977298 18235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.3:40043 every 8 connection(s)
I20260928 15:01:20.978652 18108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
I20260928 15:01:20.979544 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 18108
I20260928 15:01:20.979745 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/ts-2/wal/instance
I20260928 15:01:20.980388 31904 external_mini_cluster.cc:1428] Setting key 320040e91b0cd522febe840d575e0f85
Sep 28 15:01:20 dist-test-slave-j968 krb5kdc[17734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607680, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.3@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 15:01:21.000933 18238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:20.981433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:33817 (local address 127.31.40.62:39741)
0928 15:01:20.982050 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:20.982057 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:20.983306 (+ 1249us) server_negotiation.cc:407] Connection header received
0928 15:01:20.985328 (+ 2022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:20.985332 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:20.985408 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:20.985501 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:20.986991 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:20.988018 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:20.989493 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:20.989954 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:20.990096 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:20.994061 (+ 3965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:20.994066 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:20.994070 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:20.994106 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:20.997545 (+ 3439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:20.998341 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:20.998345 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:20.998348 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:20.998476 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:20.998971 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:20.998975 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:20.998977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:20.999264 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:20.999395 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:21.000306 (+ 911us) server_negotiation.cc:299] Negotiation successful
0928 15:01:21.000567 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":379,"threads_started":1}
I20260928 15:01:21.001891 18239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:20.982169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:39741 (local address 127.31.40.3:33817)
0928 15:01:20.983118 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:20.983176 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:20.985084 (+ 1908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:20.985704 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:20.985712 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:20.986230 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:01:20.986817 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:20.986834 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:20.988151 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:20.988155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:20.989357 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:20.989368 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:20.990215 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:20.990222 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:20.990451 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:20.991235 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:01:20.991266 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:01:20.993828 (+ 2562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:01:20.997680 (+ 3852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:01:20.997687 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:01:20.997701 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:01:20.998221 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:01:20.998614 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:01:20.998618 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:01:20.998621 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:01:20.998847 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:01:20.999399 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:01:20.999405 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:01:20.999749 (+ 344us) client_negotiation.cc:769] Sending connection context
0928 15:01:21.000176 (+ 427us) client_negotiation.cc:240] Negotiation successful
0928 15:01:21.000649 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":356,"threads_started":1}
I20260928 15:01:21.003355 18236 heartbeater.cc:347] Connected to a master server at 127.31.40.62:39741
I20260928 15:01:21.003643 18236 heartbeater.cc:464] Registering TS with master...
I20260928 15:01:21.004475 18236 heartbeater.cc:511] Master 127.31.40.62:39741 requested a full tablet report, sending...
I20260928 15:01:21.005910 17779 ts_manager.cc:194] Registered new tserver with Master: 9639a6f616c844db9095f1b2dfe9ebee (127.31.40.3:40043)
I20260928 15:01:21.006662 17779 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.3@KRBTEST.COM'} at 127.31.40.3:33817
I20260928 15:01:21.009281 31904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:01:21.026794 18238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:21.013566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36062 (local address 127.31.40.62:39741)
0928 15:01:21.013727 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:21.013732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:21.014212 (+ 480us) server_negotiation.cc:407] Connection header received
0928 15:01:21.014353 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:21.014357 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:21.014456 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:21.014537 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:21.016194 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:21.016895 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:21.018399 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:21.018795 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:21.018895 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:21.021498 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:21.021503 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:21.021506 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:21.021537 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:21.024298 (+ 2761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:21.025033 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:21.025036 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:21.025039 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:21.025150 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:21.025537 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:21.025540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:21.025541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:21.025822 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:21.026048 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:21.026372 (+ 324us) server_negotiation.cc:299] Negotiation successful
0928 15:01:21.026511 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 15:01:21.031217 17779 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36062:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:01:21.039042 17779 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:01:21.063812 18254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:21.054878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.2:35705 (local address 127.0.0.1:54910)
0928 15:01:21.056612 (+ 1734us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:21.056634 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:21.056763 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:21.057274 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:21.057278 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:21.057300 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:21.057623 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:21.057631 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:21.058945 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:21.058950 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:21.061156 (+ 2206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:21.061168 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:21.062718 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:21.062726 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:21.062914 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:21.062973 (+ 59us) client_negotiation.cc:769] Sending connection context
0928 15:01:21.063107 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 15:01:21.063297 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1609,"mutex_wait_us":100,"thread_start_us":1455,"threads_started":1}
I20260928 15:01:21.064719 18253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:21.055115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54910 (local address 127.31.40.2:35705)
0928 15:01:21.056599 (+ 1484us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:21.056607 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:21.056730 (+ 123us) server_negotiation.cc:407] Connection header received
0928 15:01:21.056944 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:21.056949 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:21.057093 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:21.057272 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:21.057764 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:21.058800 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:21.061309 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:21.062515 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:21.062652 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:21.062856 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:21.064235 (+ 1379us) server_negotiation.cc:299] Negotiation successful
0928 15:01:21.064410 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1325,"thread_start_us":1356,"threads_started":1}
I20260928 15:01:21.064759 18251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:21.052385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56162 (local address 127.31.40.3:40043)
0928 15:01:21.053570 (+ 1185us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:21.053583 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:21.054618 (+ 1035us) server_negotiation.cc:407] Connection header received
0928 15:01:21.055046 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:21.055053 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:21.055349 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:21.055532 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:21.057387 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:21.058854 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:21.061941 (+ 3087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:21.063350 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:21.063488 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:21.063710 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:21.064285 (+ 575us) server_negotiation.cc:299] Negotiation successful
0928 15:01:21.064444 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":944,"thread_start_us":736,"threads_started":1}
I20260928 15:01:21.064616 18249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:21.052406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:42919 (local address 127.0.0.1:58642)
0928 15:01:21.054614 (+ 2208us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:21.054631 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:21.054828 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:21.056051 (+ 1223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:21.056058 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:21.056095 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:21.056727 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:21.056739 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:21.059227 (+ 2488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:21.059234 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:21.061825 (+ 2591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:21.061838 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:21.063749 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:21.063756 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:21.063994 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:21.064028 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 15:01:21.064116 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 15:01:21.064240 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2093,"mutex_wait_us":59,"thread_start_us":1682,"threads_started":1}
I20260928 15:01:21.064685 18252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:21.053162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58642 (local address 127.31.40.1:42919)
0928 15:01:21.054163 (+ 1001us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:21.054174 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:21.054721 (+ 547us) server_negotiation.cc:407] Connection header received
0928 15:01:21.055235 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:21.055246 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:21.055523 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:21.055713 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:21.056876 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:21.058636 (+ 1760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:21.062028 (+ 3392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:21.063230 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:21.063361 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:21.063585 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:21.064178 (+ 593us) server_negotiation.cc:299] Negotiation successful
0928 15:01:21.064353 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":803,"thread_start_us":632,"threads_started":1}
I20260928 15:01:21.065439 18250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:21.052033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.3:40043 (local address 127.0.0.1:56162)
0928 15:01:21.054476 (+ 2443us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:21.054516 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:21.054768 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:21.055936 (+ 1168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:21.055944 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:21.055973 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:21.057137 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:21.057150 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:21.059063 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:21.059071 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:21.061763 (+ 2692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:21.061780 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:21.063893 (+ 2113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:21.063900 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:21.064082 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:21.064123 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 15:01:21.064990 (+ 867us) client_negotiation.cc:240] Negotiation successful
0928 15:01:21.065141 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2181,"mutex_wait_us":801,"thread_start_us":2316,"threads_started":1}
I20260928 15:01:21.068655 18037 tablet_service.cc:1467] Processing CreateTablet for tablet c501571dd5fb40b69e318bbca77be64f (DEFAULT_TABLE table=test-table [id=eaa614ba624a43ec8f7628b94278d349]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:01:21.068807 17903 tablet_service.cc:1467] Processing CreateTablet for tablet c501571dd5fb40b69e318bbca77be64f (DEFAULT_TABLE table=test-table [id=eaa614ba624a43ec8f7628b94278d349]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:01:21.068799 18171 tablet_service.cc:1467] Processing CreateTablet for tablet c501571dd5fb40b69e318bbca77be64f (DEFAULT_TABLE table=test-table [id=eaa614ba624a43ec8f7628b94278d349]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:01:21.069928 18037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c501571dd5fb40b69e318bbca77be64f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:21.070194 17903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c501571dd5fb40b69e318bbca77be64f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:21.070191 18171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c501571dd5fb40b69e318bbca77be64f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:21.081336 18255 tablet_bootstrap.cc:492] T c501571dd5fb40b69e318bbca77be64f P 02b3d3f838194f669e76cc67623831fa: Bootstrap starting.
I20260928 15:01:21.084594 18255 tablet_bootstrap.cc:654] T c501571dd5fb40b69e318bbca77be64f P 02b3d3f838194f669e76cc67623831fa: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:21.084805 18256 tablet_bootstrap.cc:492] T c501571dd5fb40b69e318bbca77be64f P 9639a6f616c844db9095f1b2dfe9ebee: Bootstrap starting.
I20260928 15:01:21.085791 18257 tablet_bootstrap.cc:492] T c501571dd5fb40b69e318bbca77be64f P dafb852b35dc45f0bafccad1c9f86ff9: Bootstrap starting.
I20260928 15:01:21.087559 18255 log.cc:824] T c501571dd5fb40b69e318bbca77be64f P 02b3d3f838194f669e76cc67623831fa: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:21.088572 18256 tablet_bootstrap.cc:654] T c501571dd5fb40b69e318bbca77be64f P 9639a6f616c844db9095f1b2dfe9ebee: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:21.088858 18257 tablet_bootstrap.cc:654] T c501571dd5fb40b69e318bbca77be64f P dafb852b35dc45f0bafccad1c9f86ff9: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:21.089735 18256 log.cc:824] T c501571dd5fb40b69e318bbca77be64f P 9639a6f616c844db9095f1b2dfe9ebee: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:21.089869 18257 log.cc:824] T c501571dd5fb40b69e318bbca77be64f P dafb852b35dc45f0bafccad1c9f86ff9: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:21.090811 18255 tablet_bootstrap.cc:492] T c501571dd5fb40b69e318bbca77be64f P 02b3d3f838194f669e76cc67623831fa: No bootstrap required, opened a new log
I20260928 15:01:21.091097 18255 ts_tablet_manager.cc:1397] T c501571dd5fb40b69e318bbca77be64f P 02b3d3f838194f669e76cc67623831fa: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260928 15:01:21.092505 18256 tablet_bootstrap.cc:492] T c501571dd5fb40b69e318bbca77be64f P 9639a6f616c844db9095f1b2dfe9ebee: No bootstrap required, opened a new log
I20260928 15:01:21.092767 18256 ts_tablet_manager.cc:1397] T c501571dd5fb40b69e318bbca77be64f P 9639a6f616c844db9095f1b2dfe9ebee: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260928 15:01:21.092841 18257 tablet_bootstrap.cc:492] T c501571dd5fb40b69e318bbca77be64f P dafb852b35dc45f0bafccad1c9f86ff9: No bootstrap required, opened a new log
I20260928 15:01:21.093086 18257 ts_tablet_manager.cc:1397] T c501571dd5fb40b69e318bbca77be64f P dafb852b35dc45f0bafccad1c9f86ff9: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260928 15:01:21.100284 18256 raft_consensus.cc:348] T c501571dd5fb40b69e318bbca77be64f P 9639a6f616c844db9095f1b2dfe9ebee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9639a6f616c844db9095f1b2dfe9ebee" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 40043 } } peers { permanent_uuid: "dafb852b35dc45f0bafccad1c9f86ff9" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 42919 } } peers { permanent_uuid: "02b3d3f838194f669e76cc67623831fa" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 35705 } }
I20260928 15:01:21.100642 18256 raft_consensus.cc:374] T c501571dd5fb40b69e318bbca77be64f P 9639a6f616c844db9095f1b2dfe9ebee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:21.100747 18256 raft_consensus.cc:729] T c501571dd5fb40b69e318bbca77be64f P 9639a6f616c844db9095f1b2dfe9ebee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9639a6f616c844db9095f1b2dfe9ebee, State: Initialized, Role: FOLLOWER
I20260928 15:01:21.101377 18256 consensus_queue.cc:260] T c501571dd5fb40b69e318bbca77be64f P 9639a6f616c844db9095f1b2dfe9ebee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9639a6f616c844db9095f1b2dfe9ebee" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 40043 } } peers { permanent_uuid: "dafb852b35dc45f0bafccad1c9f86ff9" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 42919 } } peers { permanent_uuid: "02b3d3f838194f669e76cc67623831fa" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 35705 } }
I20260928 15:01:21.101678 18255 raft_consensus.cc:348] T c501571dd5fb40b69e318bbca77be64f P 02b3d3f838194f669e76cc67623831fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9639a6f616c844db9095f1b2dfe9ebee" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 40043 } } peers { permanent_uuid: "dafb852b35dc45f0bafccad1c9f86ff9" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 42919 } } peers { permanent_uuid: "02b3d3f838194f669e76cc67623831fa" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 35705 } }
I20260928 15:01:21.102174 18255 raft_consensus.cc:374] T c501571dd5fb40b69e318bbca77be64f P 02b3d3f838194f669e76cc67623831fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:21.102504 18236 heartbeater.cc:503] Master 127.31.40.62:39741 was elected leader, sending a full tablet report...
I20260928 15:01:21.103029 18256 ts_tablet_manager.cc:1428] T c501571dd5fb40b69e318bbca77be64f P 9639a6f616c844db9095f1b2dfe9ebee: Time spent starting tablet: real 0.010s user 0.006s sys 0.004s
I20260928 15:01:21.103341 18257 raft_consensus.cc:348] T c501571dd5fb40b69e318bbca77be64f P dafb852b35dc45f0bafccad1c9f86ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9639a6f616c844db9095f1b2dfe9ebee" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 40043 } } peers { permanent_uuid: "dafb852b35dc45f0bafccad1c9f86ff9" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 42919 } } peers { permanent_uuid: "02b3d3f838194f669e76cc67623831fa" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 35705 } }
I20260928 15:01:21.103746 18257 raft_consensus.cc:374] T c501571dd5fb40b69e318bbca77be64f P dafb852b35dc45f0bafccad1c9f86ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:21.103854 18257 raft_consensus.cc:729] T c501571dd5fb40b69e318bbca77be64f P dafb852b35dc45f0bafccad1c9f86ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dafb852b35dc45f0bafccad1c9f86ff9, State: Initialized, Role: FOLLOWER
I20260928 15:01:21.103982 18255 raft_consensus.cc:729] T c501571dd5fb40b69e318bbca77be64f P 02b3d3f838194f669e76cc67623831fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02b3d3f838194f669e76cc67623831fa, State: Initialized, Role: FOLLOWER
I20260928 15:01:21.104506 18257 consensus_queue.cc:260] T c501571dd5fb40b69e318bbca77be64f P dafb852b35dc45f0bafccad1c9f86ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9639a6f616c844db9095f1b2dfe9ebee" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 40043 } } peers { permanent_uuid: "dafb852b35dc45f0bafccad1c9f86ff9" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 42919 } } peers { permanent_uuid: "02b3d3f838194f669e76cc67623831fa" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 35705 } }
I20260928 15:01:21.104662 18255 consensus_queue.cc:260] T c501571dd5fb40b69e318bbca77be64f P 02b3d3f838194f669e76cc67623831fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9639a6f616c844db9095f1b2dfe9ebee" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 40043 } } peers { permanent_uuid: "dafb852b35dc45f0bafccad1c9f86ff9" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 42919 } } peers { permanent_uuid: "02b3d3f838194f669e76cc67623831fa" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 35705 } }
I20260928 15:01:21.108461 18102 heartbeater.cc:503] Master 127.31.40.62:39741 was elected leader, sending a full tablet report...
I20260928 15:01:21.109181 18255 ts_tablet_manager.cc:1428] T c501571dd5fb40b69e318bbca77be64f P 02b3d3f838194f669e76cc67623831fa: Time spent starting tablet: real 0.018s user 0.014s sys 0.003s
I20260928 15:01:21.110586 18257 ts_tablet_manager.cc:1428] T c501571dd5fb40b69e318bbca77be64f P dafb852b35dc45f0bafccad1c9f86ff9: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s
W20260928 15:01:21.166392 17969 tablet.cc:2369] T c501571dd5fb40b69e318bbca77be64f P dafb852b35dc45f0bafccad1c9f86ff9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:01:21.175143 18262 raft_consensus.cc:482] T c501571dd5fb40b69e318bbca77be64f P dafb852b35dc45f0bafccad1c9f86ff9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:01:21.175439 18262 raft_consensus.cc:504] T c501571dd5fb40b69e318bbca77be64f P dafb852b35dc45f0bafccad1c9f86ff9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9639a6f616c844db9095f1b2dfe9ebee" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 40043 } } peers { permanent_uuid: "dafb852b35dc45f0bafccad1c9f86ff9" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 42919 } } peers { permanent_uuid: "02b3d3f838194f669e76cc67623831fa" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 35705 } }
I20260928 15:01:21.177201 18262 leader_election.cc:288] T c501571dd5fb40b69e318bbca77be64f P dafb852b35dc45f0bafccad1c9f86ff9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9639a6f616c844db9095f1b2dfe9ebee (127.31.40.3:40043), 02b3d3f838194f669e76cc67623831fa (127.31.40.2:35705)
I20260928 15:01:21.184303 18251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:21.177671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:48393 (local address 127.31.40.3:40043)
0928 15:01:21.177946 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:21.177955 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:21.178395 (+ 440us) server_negotiation.cc:407] Connection header received
0928 15:01:21.178685 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:21.178691 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:21.178841 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:21.178917 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:21.179845 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:21.180743 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:21.182505 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:21.183328 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:21.183480 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:21.183590 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:21.183898 (+ 308us) server_negotiation.cc:299] Negotiation successful
0928 15:01:21.184029 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 15:01:21.184357 18264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:21.177608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.3:40043 (local address 127.31.40.1:48393)
0928 15:01:21.178213 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:21.178230 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:21.178451 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:21.179141 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:21.179145 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:21.179172 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:21.179671 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:21.179685 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:21.180880 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:21.180886 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:21.182350 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:21.182361 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:21.183536 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:21.183542 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:21.183704 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:21.183735 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 15:01:21.183841 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 15:01:21.183992 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":332,"threads_started":1}
I20260928 15:01:21.185170 18253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:21.178892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:34443 (local address 127.31.40.2:35705)
0928 15:01:21.179160 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:21.179169 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:21.179888 (+ 719us) server_negotiation.cc:407] Connection header received
0928 15:01:21.180075 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:21.180081 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:21.180160 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:21.180288 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:21.180833 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:21.181505 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:21.183214 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:21.184065 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:21.184210 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:21.184319 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:21.184780 (+ 461us) server_negotiation.cc:299] Negotiation successful
0928 15:01:21.184910 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260928 15:01:21.185230 18265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:21.178373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.2:35705 (local address 127.31.40.1:34443)
0928 15:01:21.179739 (+ 1366us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:21.179783 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:21.179940 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:21.180292 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:21.180296 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:21.180319 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:21.180656 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:21.180665 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:21.181647 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:21.181654 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:21.183046 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:21.183058 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:21.184292 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:21.184298 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:21.184626 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:21.184655 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 15:01:21.184738 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 15:01:21.184911 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1261,"thread_start_us":1012,"threads_started":1}
I20260928 15:01:21.185295 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c501571dd5fb40b69e318bbca77be64f" candidate_uuid: "dafb852b35dc45f0bafccad1c9f86ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9639a6f616c844db9095f1b2dfe9ebee" is_pre_election: true
I20260928 15:01:21.186000 18191 raft_consensus.cc:2454] T c501571dd5fb40b69e318bbca77be64f P 9639a6f616c844db9095f1b2dfe9ebee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dafb852b35dc45f0bafccad1c9f86ff9 in term 0.
I20260928 15:01:21.186349 18057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c501571dd5fb40b69e318bbca77be64f" candidate_uuid: "dafb852b35dc45f0bafccad1c9f86ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02b3d3f838194f669e76cc67623831fa" is_pre_election: true
I20260928 15:01:21.186811 17857 leader_election.cc:302] T c501571dd5fb40b69e318bbca77be64f P dafb852b35dc45f0bafccad1c9f86ff9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9639a6f616c844db9095f1b2dfe9ebee, dafb852b35dc45f0bafccad1c9f86ff9; no voters:
I20260928 15:01:21.186875 18057 raft_consensus.cc:2454] T c501571dd5fb40b69e318bbca77be64f P 02b3d3f838194f669e76cc67623831fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dafb852b35dc45f0bafccad1c9f86ff9 in term 0.
I20260928 15:01:21.187243 18262 raft_consensus.cc:2781] T c501571dd5fb40b69e318bbca77be64f P dafb852b35dc45f0bafccad1c9f86ff9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:01:21.187469 18262 raft_consensus.cc:482] T c501571dd5fb40b69e318bbca77be64f P dafb852b35dc45f0bafccad1c9f86ff9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:01:21.187582 18262 raft_consensus.cc:3037] T c501571dd5fb40b69e318bbca77be64f P dafb852b35dc45f0bafccad1c9f86ff9 [term 0 FOLLOWER]: Advancing to term 1
W20260928 15:01:21.187780 18103 tablet.cc:2369] T c501571dd5fb40b69e318bbca77be64f P 02b3d3f838194f669e76cc67623831fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:01:21.189857 18262 raft_consensus.cc:504] T c501571dd5fb40b69e318bbca77be64f P dafb852b35dc45f0bafccad1c9f86ff9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9639a6f616c844db9095f1b2dfe9ebee" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 40043 } } peers { permanent_uuid: "dafb852b35dc45f0bafccad1c9f86ff9" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 42919 } } peers { permanent_uuid: "02b3d3f838194f669e76cc67623831fa" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 35705 } }
I20260928 15:01:21.190490 18262 leader_election.cc:288] T c501571dd5fb40b69e318bbca77be64f P dafb852b35dc45f0bafccad1c9f86ff9 [CANDIDATE]: Term 1 election: Requested vote from peers 9639a6f616c844db9095f1b2dfe9ebee (127.31.40.3:40043), 02b3d3f838194f669e76cc67623831fa (127.31.40.2:35705)
I20260928 15:01:21.190963 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c501571dd5fb40b69e318bbca77be64f" candidate_uuid: "dafb852b35dc45f0bafccad1c9f86ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9639a6f616c844db9095f1b2dfe9ebee"
I20260928 15:01:21.190997 18057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c501571dd5fb40b69e318bbca77be64f" candidate_uuid: "dafb852b35dc45f0bafccad1c9f86ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02b3d3f838194f669e76cc67623831fa"
I20260928 15:01:21.191195 18057 raft_consensus.cc:3037] T c501571dd5fb40b69e318bbca77be64f P 02b3d3f838194f669e76cc67623831fa [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:21.191172 18191 raft_consensus.cc:3037] T c501571dd5fb40b69e318bbca77be64f P 9639a6f616c844db9095f1b2dfe9ebee [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:21.193389 18057 raft_consensus.cc:2454] T c501571dd5fb40b69e318bbca77be64f P 02b3d3f838194f669e76cc67623831fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dafb852b35dc45f0bafccad1c9f86ff9 in term 1.
I20260928 15:01:21.193413 18191 raft_consensus.cc:2454] T c501571dd5fb40b69e318bbca77be64f P 9639a6f616c844db9095f1b2dfe9ebee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dafb852b35dc45f0bafccad1c9f86ff9 in term 1.
I20260928 15:01:21.193784 17857 leader_election.cc:302] T c501571dd5fb40b69e318bbca77be64f P dafb852b35dc45f0bafccad1c9f86ff9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02b3d3f838194f669e76cc67623831fa, dafb852b35dc45f0bafccad1c9f86ff9; no voters:
I20260928 15:01:21.194099 18262 raft_consensus.cc:2781] T c501571dd5fb40b69e318bbca77be64f P dafb852b35dc45f0bafccad1c9f86ff9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:01:21.194540 18262 raft_consensus.cc:686] T c501571dd5fb40b69e318bbca77be64f P dafb852b35dc45f0bafccad1c9f86ff9 [term 1 LEADER]: Becoming Leader. State: Replica: dafb852b35dc45f0bafccad1c9f86ff9, State: Running, Role: LEADER
I20260928 15:01:21.195384 18262 consensus_queue.cc:237] T c501571dd5fb40b69e318bbca77be64f P dafb852b35dc45f0bafccad1c9f86ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9639a6f616c844db9095f1b2dfe9ebee" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 40043 } } peers { permanent_uuid: "dafb852b35dc45f0bafccad1c9f86ff9" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 42919 } } peers { permanent_uuid: "02b3d3f838194f669e76cc67623831fa" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 35705 } }
I20260928 15:01:21.199926 17778 catalog_manager.cc:5495] T c501571dd5fb40b69e318bbca77be64f P dafb852b35dc45f0bafccad1c9f86ff9 reported cstate change: term changed from 0 to 1, leader changed from <none> to dafb852b35dc45f0bafccad1c9f86ff9 (127.31.40.1). New cstate: current_term: 1 leader_uuid: "dafb852b35dc45f0bafccad1c9f86ff9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9639a6f616c844db9095f1b2dfe9ebee" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 40043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dafb852b35dc45f0bafccad1c9f86ff9" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 42919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02b3d3f838194f669e76cc67623831fa" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 35705 } health_report { overall_health: UNKNOWN } } }
I20260928 15:01:21.223297 18252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:21.217160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58652 (local address 127.31.40.1:42919)
0928 15:01:21.217350 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:21.217355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:21.217372 (+ 17us) server_negotiation.cc:407] Connection header received
0928 15:01:21.217611 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:21.217616 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:21.217704 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:21.217845 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:01:21.218330 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:21.219157 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:21.220758 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:21.221194 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:21.221297 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:21.221999 (+ 702us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:01:21.222670 (+ 671us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:01:21.222814 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:21.223030 (+ 216us) server_negotiation.cc:299] Negotiation successful
0928 15:01:21.223075 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260928 15:01:21.230402 18237 tablet.cc:2369] T c501571dd5fb40b69e318bbca77be64f P 9639a6f616c844db9095f1b2dfe9ebee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:01:21.232206 18191 raft_consensus.cc:1261] T c501571dd5fb40b69e318bbca77be64f P 9639a6f616c844db9095f1b2dfe9ebee [term 1 FOLLOWER]: Refusing update from remote peer dafb852b35dc45f0bafccad1c9f86ff9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:01:21.232453 18057 raft_consensus.cc:1261] T c501571dd5fb40b69e318bbca77be64f P 02b3d3f838194f669e76cc67623831fa [term 1 FOLLOWER]: Refusing update from remote peer dafb852b35dc45f0bafccad1c9f86ff9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:01:21.233140 18262 consensus_queue.cc:1035] T c501571dd5fb40b69e318bbca77be64f P dafb852b35dc45f0bafccad1c9f86ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9639a6f616c844db9095f1b2dfe9ebee" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 40043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:01:21.233466 18266 consensus_queue.cc:1035] T c501571dd5fb40b69e318bbca77be64f P dafb852b35dc45f0bafccad1c9f86ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02b3d3f838194f669e76cc67623831fa" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 35705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:01:21.245345 18269 mvcc.cc:205] Tried to move back new op lower bound from 7334329062316359680 to 7334329062180294656. Current Snapshot: MvccSnapshot[applied={T|T < 7334329062316359680}]
I20260928 15:01:21.259370 17779 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:01:21.259647 17779 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:01:21.263051 17779 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37d0a56e7cba49a8b0bfdc6af53f031c: Sending DeleteTablet for 3 replicas of tablet c501571dd5fb40b69e318bbca77be64f
I20260928 15:01:21.264109 18171 tablet_service.cc:1514] Processing DeleteTablet for tablet c501571dd5fb40b69e318bbca77be64f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:01:21 UTC) from {username='kudu', principal='kudu/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:56162
I20260928 15:01:21.264220 17903 tablet_service.cc:1514] Processing DeleteTablet for tablet c501571dd5fb40b69e318bbca77be64f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:01:21 UTC) from {username='kudu', principal='kudu/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:58642
I20260928 15:01:21.267042 18037 tablet_service.cc:1514] Processing DeleteTablet for tablet c501571dd5fb40b69e318bbca77be64f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:01:21 UTC) from {username='kudu', principal='kudu/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:54910
I20260928 15:01:21.268460 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 17840
I20260928 15:01:21.271255 18281 tablet_replica.cc:330] T c501571dd5fb40b69e318bbca77be64f P 02b3d3f838194f669e76cc67623831fa: stopping tablet replica
I20260928 15:01:21.271781 18281 raft_consensus.cc:2229] T c501571dd5fb40b69e318bbca77be64f P 02b3d3f838194f669e76cc67623831fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:01:21.272383 18281 raft_consensus.cc:2258] T c501571dd5fb40b69e318bbca77be64f P 02b3d3f838194f669e76cc67623831fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:01:21.273403 18280 tablet_replica.cc:330] T c501571dd5fb40b69e318bbca77be64f P dafb852b35dc45f0bafccad1c9f86ff9: stopping tablet replica
I20260928 15:01:21.273904 18280 raft_consensus.cc:2229] T c501571dd5fb40b69e318bbca77be64f P dafb852b35dc45f0bafccad1c9f86ff9 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:01:21.274683 18280 raft_consensus.cc:2258] T c501571dd5fb40b69e318bbca77be64f P dafb852b35dc45f0bafccad1c9f86ff9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:01:21.275767 18279 tablet_replica.cc:330] T c501571dd5fb40b69e318bbca77be64f P 9639a6f616c844db9095f1b2dfe9ebee: stopping tablet replica
I20260928 15:01:21.276165 18279 raft_consensus.cc:2229] T c501571dd5fb40b69e318bbca77be64f P 9639a6f616c844db9095f1b2dfe9ebee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:01:21.276645 18279 raft_consensus.cc:2258] T c501571dd5fb40b69e318bbca77be64f P 9639a6f616c844db9095f1b2dfe9ebee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:01:21.277004 18280 ts_tablet_manager.cc:1905] T c501571dd5fb40b69e318bbca77be64f P dafb852b35dc45f0bafccad1c9f86ff9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:01:21.279243 18279 ts_tablet_manager.cc:1905] T c501571dd5fb40b69e318bbca77be64f P 9639a6f616c844db9095f1b2dfe9ebee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:01:21.279282 18281 ts_tablet_manager.cc:1905] T c501571dd5fb40b69e318bbca77be64f P 02b3d3f838194f669e76cc67623831fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:01:21.284164 18280 ts_tablet_manager.cc:1918] T c501571dd5fb40b69e318bbca77be64f P dafb852b35dc45f0bafccad1c9f86ff9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:01:21.284372 18280 log.cc:1196] T c501571dd5fb40b69e318bbca77be64f P dafb852b35dc45f0bafccad1c9f86ff9: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/ts-0/wal/wals/c501571dd5fb40b69e318bbca77be64f
I20260928 15:01:21.284900 18280 ts_tablet_manager.cc:1939] T c501571dd5fb40b69e318bbca77be64f P dafb852b35dc45f0bafccad1c9f86ff9: Deleting consensus metadata
I20260928 15:01:21.286576 18279 ts_tablet_manager.cc:1918] T c501571dd5fb40b69e318bbca77be64f P 9639a6f616c844db9095f1b2dfe9ebee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:01:21.286813 18279 log.cc:1196] T c501571dd5fb40b69e318bbca77be64f P 9639a6f616c844db9095f1b2dfe9ebee: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/ts-2/wal/wals/c501571dd5fb40b69e318bbca77be64f
I20260928 15:01:21.286839 17766 catalog_manager.cc:4841] TS dafb852b35dc45f0bafccad1c9f86ff9 (127.31.40.1:42919): tablet c501571dd5fb40b69e318bbca77be64f (table test-table [id=eaa614ba624a43ec8f7628b94278d349]) successfully deleted
I20260928 15:01:21.287400 18279 ts_tablet_manager.cc:1939] T c501571dd5fb40b69e318bbca77be64f P 9639a6f616c844db9095f1b2dfe9ebee: Deleting consensus metadata
I20260928 15:01:21.289342 17764 catalog_manager.cc:4841] TS 9639a6f616c844db9095f1b2dfe9ebee (127.31.40.3:40043): tablet c501571dd5fb40b69e318bbca77be64f (table test-table [id=eaa614ba624a43ec8f7628b94278d349]) successfully deleted
I20260928 15:01:21.289681 18281 ts_tablet_manager.cc:1918] T c501571dd5fb40b69e318bbca77be64f P 02b3d3f838194f669e76cc67623831fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:01:21.289866 18281 log.cc:1196] T c501571dd5fb40b69e318bbca77be64f P 02b3d3f838194f669e76cc67623831fa: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607365362981-31904-0/minicluster-data/ts-1/wal/wals/c501571dd5fb40b69e318bbca77be64f
I20260928 15:01:21.290419 18281 ts_tablet_manager.cc:1939] T c501571dd5fb40b69e318bbca77be64f P 02b3d3f838194f669e76cc67623831fa: Deleting consensus metadata
I20260928 15:01:21.292169 17764 catalog_manager.cc:4841] TS 02b3d3f838194f669e76cc67623831fa (127.31.40.2:35705): tablet c501571dd5fb40b69e318bbca77be64f (table test-table [id=eaa614ba624a43ec8f7628b94278d349]) successfully deleted
Sep 28 15:01:21 dist-test-slave-j968 krb5kdc[17734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.1@KRBTEST.COM
I20260928 15:01:21.304270 18252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:21.269715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58668 (local address 127.31.40.1:42919)
0928 15:01:21.270432 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:21.270438 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:21.277495 (+ 7057us) server_negotiation.cc:407] Connection header received
0928 15:01:21.277804 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:21.277811 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:21.277911 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:21.277987 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:21.285866 (+ 7879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:21.286991 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:21.289018 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:21.289737 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:21.289843 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:21.296545 (+ 6702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:21.296560 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:21.296583 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:21.296642 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:21.300780 (+ 4138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:21.301780 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:21.301790 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:21.301794 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:21.301989 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:21.302469 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:21.302475 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:21.302478 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:21.302939 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:21.303239 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:21.303567 (+ 328us) server_negotiation.cc:299] Negotiation successful
0928 15:01:21.303829 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577}
I20260928 15:01:21.305023 17963 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:01:21.404675 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 17840
I20260928 15:01:21.424628 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 17974
Sep 28 15:01:21 dist-test-slave-j968 krb5kdc[17734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.2@KRBTEST.COM
I20260928 15:01:21.441083 18253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:21.425573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54914 (local address 127.31.40.2:35705)
0928 15:01:21.425757 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:21.425763 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:21.425786 (+ 23us) server_negotiation.cc:407] Connection header received
0928 15:01:21.425946 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:21.425950 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:21.426018 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:21.426105 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:21.427486 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:21.428403 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:21.429669 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:21.430141 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:21.430270 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:21.434687 (+ 4417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:21.434702 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:21.434726 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:21.434766 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:21.437949 (+ 3183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:21.438739 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:21.438749 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:21.438753 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:21.438910 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:21.439289 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:21.439299 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:21.439303 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:21.439826 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:21.440153 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:21.440486 (+ 333us) server_negotiation.cc:299] Negotiation successful
0928 15:01:21.440709 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 15:01:21.441860 18097 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:01:21.534196 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 17974
I20260928 15:01:21.549595 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 18108
Sep 28 15:01:21 dist-test-slave-j968 krb5kdc[17734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.3@KRBTEST.COM
I20260928 15:01:21.565986 18251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:21.550496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56170 (local address 127.31.40.3:40043)
0928 15:01:21.550709 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:21.550714 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:21.550737 (+ 23us) server_negotiation.cc:407] Connection header received
0928 15:01:21.550886 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:21.550891 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:21.550978 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:21.551091 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:21.552767 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:21.553553 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:21.554839 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:21.555257 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:21.555402 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:21.560029 (+ 4627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:21.560046 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:21.560063 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:21.560107 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:21.563000 (+ 2893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:21.563738 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:21.563745 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:21.563747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:21.563955 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:21.564379 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:21.564385 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:21.564388 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:21.564764 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:21.565062 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:21.565407 (+ 345us) server_negotiation.cc:299] Negotiation successful
0928 15:01:21.565643 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 15:01:21.566725 18231 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:01:21.658221 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 18108
I20260928 15:01:21.675797 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 17747
I20260928 15:01:21.677030 17809 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:01:21.791572 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 17747
2026-09-28T15:01:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4340 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T15:01:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:01:21Z Disabled control of system clock
I20260928 15:01:21.873713 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:40205
--webserver_interface=127.31.40.62
--webserver_port=0
--builtin_ntp_servers=127.31.40.20:32833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.40.62:40205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 15:01:22.220644 18290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:22.220973 18290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:22.221060 18290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:22.228356 18290 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:01:22.228468 18290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:22.228506 18290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:22.228556 18290 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:01:22.228600 18290 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:01:22.238044 18290 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:32833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.40.62:40205
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:40205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:01:22.239353 18290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:22.241163 18290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:22.253331 18298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:22.253376 18295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:22.253409 18296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:22.254338 18290 server_base.cc:1034] running on GCE node
I20260928 15:01:22.255155 18290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:22.256695 18290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:22.257907 18290 hybrid_clock.cc:648] HybridClock initialized: now 1790607682257837 us; error 60 us; skew 500 ppm
I20260928 15:01:22.258378 18290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:01:22.272634 18290 webserver.cc:460] Webserver started at http://127.31.40.62:37429/ using document root <none> and password file <none>
I20260928 15:01:22.273432 18290 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:22.273553 18290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:22.273921 18290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:22.277081 18290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/master-0/data/instance:
uuid: "17d16212cd294f0fbdf5904bf61d6c8a"
format_stamp: "Formatted at 2026-09-28 15:01:22 on dist-test-slave-j968"
server_key: "db1e2e9fd1a91e71ca4f102f96af639f"
server_key_iv: "a0e8947685d0eaecdbd4b1d8188358e5"
server_key_version: "encryptionkey@0"
I20260928 15:01:22.278002 18290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/master-0/wal/instance:
uuid: "17d16212cd294f0fbdf5904bf61d6c8a"
format_stamp: "Formatted at 2026-09-28 15:01:22 on dist-test-slave-j968"
server_key: "db1e2e9fd1a91e71ca4f102f96af639f"
server_key_iv: "a0e8947685d0eaecdbd4b1d8188358e5"
server_key_version: "encryptionkey@0"
I20260928 15:01:22.283366 18290 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 15:01:22.287145 18304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:22.288731 18290 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:01:22.288931 18290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/master-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/master-0/wal
uuid: "17d16212cd294f0fbdf5904bf61d6c8a"
format_stamp: "Formatted at 2026-09-28 15:01:22 on dist-test-slave-j968"
server_key: "db1e2e9fd1a91e71ca4f102f96af639f"
server_key_iv: "a0e8947685d0eaecdbd4b1d8188358e5"
server_key_version: "encryptionkey@0"
I20260928 15:01:22.289162 18290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:22.317010 18290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:22.317929 18290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:22.318223 18290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:22.337457 18290 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.62:40205
I20260928 15:01:22.337481 18355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.62:40205 every 8 connection(s)
I20260928 15:01:22.338946 18290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
I20260928 15:01:22.339437 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 18290
I20260928 15:01:22.339628 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/master-0/wal/instance
I20260928 15:01:22.340334 31904 external_mini_cluster.cc:1428] Setting key f13404b5fb83345be0653a05bc8549b5
I20260928 15:01:22.343798 18356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:22.356609 18359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:22.342362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52226 (local address 127.31.40.62:40205)
0928 15:01:22.343428 (+ 1066us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:22.343443 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:22.343493 (+ 50us) server_negotiation.cc:407] Connection header received
0928 15:01:22.345976 (+ 2483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:22.346005 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:22.346598 (+ 593us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:22.347071 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:22.347620 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:22.349690 (+ 2070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:22.351186 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:22.352241 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:22.352429 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:22.353628 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:22.353638 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:22.353661 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:01:22.354425 (+ 764us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:22.354534 (+ 109us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:01:22.354566 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:22.354683 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:22.354930 (+ 247us) server_negotiation.cc:299] Negotiation successful
0928 15:01:22.355293 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":336,"threads_started":1}
I20260928 15:01:22.358107 18356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17d16212cd294f0fbdf5904bf61d6c8a: Bootstrap starting.
I20260928 15:01:22.361544 18356 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17d16212cd294f0fbdf5904bf61d6c8a: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:22.362586 18356 log.cc:824] T 00000000000000000000000000000000 P 17d16212cd294f0fbdf5904bf61d6c8a: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:22.365736 18356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17d16212cd294f0fbdf5904bf61d6c8a: No bootstrap required, opened a new log
I20260928 15:01:22.373265 18356 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17d16212cd294f0fbdf5904bf61d6c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17d16212cd294f0fbdf5904bf61d6c8a" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 40205 } }
I20260928 15:01:22.373613 18356 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17d16212cd294f0fbdf5904bf61d6c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:22.373685 18356 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17d16212cd294f0fbdf5904bf61d6c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17d16212cd294f0fbdf5904bf61d6c8a, State: Initialized, Role: FOLLOWER
I20260928 15:01:22.374163 18356 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17d16212cd294f0fbdf5904bf61d6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17d16212cd294f0fbdf5904bf61d6c8a" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 40205 } }
I20260928 15:01:22.374358 18356 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17d16212cd294f0fbdf5904bf61d6c8a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:01:22.374449 18356 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17d16212cd294f0fbdf5904bf61d6c8a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:01:22.374568 18356 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17d16212cd294f0fbdf5904bf61d6c8a [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:22.376782 18356 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17d16212cd294f0fbdf5904bf61d6c8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17d16212cd294f0fbdf5904bf61d6c8a" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 40205 } }
I20260928 15:01:22.377245 18356 leader_election.cc:302] T 00000000000000000000000000000000 P 17d16212cd294f0fbdf5904bf61d6c8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17d16212cd294f0fbdf5904bf61d6c8a; no voters:
I20260928 15:01:22.377807 18356 leader_election.cc:288] T 00000000000000000000000000000000 P 17d16212cd294f0fbdf5904bf61d6c8a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:01:22.377942 18361 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17d16212cd294f0fbdf5904bf61d6c8a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:01:22.379357 18361 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17d16212cd294f0fbdf5904bf61d6c8a [term 1 LEADER]: Becoming Leader. State: Replica: 17d16212cd294f0fbdf5904bf61d6c8a, State: Running, Role: LEADER
I20260928 15:01:22.379923 18361 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17d16212cd294f0fbdf5904bf61d6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17d16212cd294f0fbdf5904bf61d6c8a" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 40205 } }
I20260928 15:01:22.380117 18356 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17d16212cd294f0fbdf5904bf61d6c8a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:01:22.382943 18363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17d16212cd294f0fbdf5904bf61d6c8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17d16212cd294f0fbdf5904bf61d6c8a. Latest consensus state: current_term: 1 leader_uuid: "17d16212cd294f0fbdf5904bf61d6c8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17d16212cd294f0fbdf5904bf61d6c8a" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 40205 } } }
I20260928 15:01:22.382937 18362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17d16212cd294f0fbdf5904bf61d6c8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17d16212cd294f0fbdf5904bf61d6c8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17d16212cd294f0fbdf5904bf61d6c8a" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 40205 } } }
I20260928 15:01:22.383452 18363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17d16212cd294f0fbdf5904bf61d6c8a [sys.catalog]: This master's current role is: LEADER
I20260928 15:01:22.383450 18362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17d16212cd294f0fbdf5904bf61d6c8a [sys.catalog]: This master's current role is: LEADER
I20260928 15:01:22.385069 18369 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:01:22.390852 18369 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:01:22.400599 18369 catalog_manager.cc:1347] Generated new cluster ID: 51be89e3b58943389aa1f515faaa7e2d
I20260928 15:01:22.400720 18369 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:01:22.418829 18369 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:01:22.420408 18369 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:01:22.433504 18369 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17d16212cd294f0fbdf5904bf61d6c8a: Generated new TSK 0
I20260928 15:01:22.434633 18369 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:01:22.446446 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.1:0
--local_ip_for_outbound_sockets=127.31.40.1
--webserver_interface=127.31.40.1
--webserver_port=0
--tserver_master_addrs=127.31.40.62:40205
--builtin_ntp_servers=127.31.40.20:32833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 15:01:22.800377 18380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:22.800668 18380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:22.800838 18380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:22.808225 18380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:22.808333 18380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:22.808533 18380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.1
I20260928 15:01:22.817842 18380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:32833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.40.1
--webserver_port=0
--tserver_master_addrs=127.31.40.62:40205
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.1
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:01:22.819181 18380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:22.821039 18380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:22.835166 18385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:22.835095 18388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:22.835124 18386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:22.835773 18380 server_base.cc:1034] running on GCE node
I20260928 15:01:22.836586 18380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:22.837634 18380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:22.838881 18380 hybrid_clock.cc:648] HybridClock initialized: now 1790607682838828 us; error 57 us; skew 500 ppm
I20260928 15:01:22.839227 18380 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:01:22.841732 18380 webserver.cc:460] Webserver started at http://127.31.40.1:39589/ using document root <none> and password file <none>
I20260928 15:01:22.842478 18380 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:22.842623 18380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:22.842955 18380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:22.845995 18380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/ts-0/data/instance:
uuid: "0ad302c7657f4607b1c4d4cf61aed260"
format_stamp: "Formatted at 2026-09-28 15:01:22 on dist-test-slave-j968"
server_key: "89e39f239efc9e1b7a14ae349a1e6360"
server_key_iv: "772e41ca7798b31ba8680f6084f6d020"
server_key_version: "encryptionkey@0"
I20260928 15:01:22.846818 18380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/ts-0/wal/instance:
uuid: "0ad302c7657f4607b1c4d4cf61aed260"
format_stamp: "Formatted at 2026-09-28 15:01:22 on dist-test-slave-j968"
server_key: "89e39f239efc9e1b7a14ae349a1e6360"
server_key_iv: "772e41ca7798b31ba8680f6084f6d020"
server_key_version: "encryptionkey@0"
I20260928 15:01:22.852267 18380 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 15:01:22.856118 18394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:22.857616 18380 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:01:22.857793 18380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/ts-0/wal
uuid: "0ad302c7657f4607b1c4d4cf61aed260"
format_stamp: "Formatted at 2026-09-28 15:01:22 on dist-test-slave-j968"
server_key: "89e39f239efc9e1b7a14ae349a1e6360"
server_key_iv: "772e41ca7798b31ba8680f6084f6d020"
server_key_version: "encryptionkey@0"
I20260928 15:01:22.858016 18380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:22.878402 18380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:22.879290 18380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:22.879592 18380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:22.880789 18380 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:01:22.882464 18380 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:01:22.882596 18380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:22.882722 18380 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:01:22.882812 18380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:22.917335 18380 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.1:40577
I20260928 15:01:22.917444 18506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.1:40577 every 8 connection(s)
I20260928 15:01:22.918699 18380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
I20260928 15:01:22.926016 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 18380
I20260928 15:01:22.926234 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/ts-0/wal/instance
I20260928 15:01:22.926783 31904 external_mini_cluster.cc:1428] Setting key a3c9b509b4d6b431503e841eb034494a
I20260928 15:01:22.929761 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.2:0
--local_ip_for_outbound_sockets=127.31.40.2
--webserver_interface=127.31.40.2
--webserver_port=0
--tserver_master_addrs=127.31.40.62:40205
--builtin_ntp_servers=127.31.40.20:32833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 15:01:22.932538 18509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:22.921681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:52199 (local address 127.31.40.62:40205)
0928 15:01:22.922264 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:22.922271 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:22.923454 (+ 1183us) server_negotiation.cc:407] Connection header received
0928 15:01:22.925310 (+ 1856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:22.925315 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:22.925383 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:22.925493 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:22.926728 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:22.927775 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:22.929533 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:22.930033 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:22.930175 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:22.931596 (+ 1421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:22.931601 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:22.931604 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:01:22.931623 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:22.931664 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:01:22.931676 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:22.931771 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:22.932061 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 15:01:22.932157 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":250,"threads_started":1}
I20260928 15:01:22.933818 18510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:22.922276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40205 (local address 127.31.40.1:52199)
0928 15:01:22.923294 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:22.923336 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:22.925036 (+ 1700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:22.925687 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:22.925694 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:22.925853 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:01:22.926546 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:22.926565 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:22.927990 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:22.927994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:22.929369 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:22.929381 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:22.930268 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:22.930278 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:22.930521 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:22.931277 (+ 756us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:01:22.931279 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:01:22.931318 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:01:22.931402 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:01:22.931771 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:01:22.931785 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:01:22.931796 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 15:01:22.932052 (+ 256us) client_negotiation.cc:240] Negotiation successful
0928 15:01:22.932460 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":281,"threads_started":1}
I20260928 15:01:22.935384 18507 heartbeater.cc:347] Connected to a master server at 127.31.40.62:40205
I20260928 15:01:22.935731 18507 heartbeater.cc:464] Registering TS with master...
I20260928 15:01:22.936743 18507 heartbeater.cc:511] Master 127.31.40.62:40205 requested a full tablet report, sending...
I20260928 15:01:22.939127 18321 ts_manager.cc:194] Registered new tserver with Master: 0ad302c7657f4607b1c4d4cf61aed260 (127.31.40.1:40577)
I20260928 15:01:22.940663 18321 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.40.1:52199
W20260928 15:01:23.269408 18511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:23.269685 18511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:23.269819 18511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:23.277266 18511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:23.277367 18511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:23.277530 18511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.2
I20260928 15:01:23.287914 18511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:32833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.40.2
--webserver_port=0
--tserver_master_addrs=127.31.40.62:40205
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.2
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:01:23.289294 18511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:23.291100 18511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:23.305255 18517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:23.305660 18516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:23.305707 18519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:23.306679 18511 server_base.cc:1034] running on GCE node
I20260928 15:01:23.307317 18511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:23.308403 18511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:23.309638 18511 hybrid_clock.cc:648] HybridClock initialized: now 1790607683309585 us; error 67 us; skew 500 ppm
I20260928 15:01:23.309974 18511 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:01:23.312366 18511 webserver.cc:460] Webserver started at http://127.31.40.2:42447/ using document root <none> and password file <none>
I20260928 15:01:23.313072 18511 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:23.313202 18511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:23.313527 18511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:23.316563 18511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/ts-1/data/instance:
uuid: "f314588e5b6445c7a4bc17c9af2db7cb"
format_stamp: "Formatted at 2026-09-28 15:01:23 on dist-test-slave-j968"
server_key: "e290d8316a9529f3310528cb876400d2"
server_key_iv: "30744b220da7f40545318aca2dd84b2e"
server_key_version: "encryptionkey@0"
I20260928 15:01:23.317338 18511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/ts-1/wal/instance:
uuid: "f314588e5b6445c7a4bc17c9af2db7cb"
format_stamp: "Formatted at 2026-09-28 15:01:23 on dist-test-slave-j968"
server_key: "e290d8316a9529f3310528cb876400d2"
server_key_iv: "30744b220da7f40545318aca2dd84b2e"
server_key_version: "encryptionkey@0"
I20260928 15:01:23.322638 18511 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 15:01:23.326373 18525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:23.327831 18511 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260928 15:01:23.328047 18511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/ts-1/wal
uuid: "f314588e5b6445c7a4bc17c9af2db7cb"
format_stamp: "Formatted at 2026-09-28 15:01:23 on dist-test-slave-j968"
server_key: "e290d8316a9529f3310528cb876400d2"
server_key_iv: "30744b220da7f40545318aca2dd84b2e"
server_key_version: "encryptionkey@0"
I20260928 15:01:23.328238 18511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:23.342141 18511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:23.342999 18511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:23.343286 18511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:23.344460 18511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:01:23.345969 18511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:01:23.346055 18511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:23.346202 18511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:01:23.346288 18511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:23.380858 18511 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.2:34177
I20260928 15:01:23.380903 18637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.2:34177 every 8 connection(s)
I20260928 15:01:23.382206 18511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
I20260928 15:01:23.387725 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 18511
I20260928 15:01:23.387913 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/ts-1/wal/instance
I20260928 15:01:23.388368 31904 external_mini_cluster.cc:1428] Setting key c8baf21b40bf03d91b2f02e1ad4e2af8
I20260928 15:01:23.391444 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.3:0
--local_ip_for_outbound_sockets=127.31.40.3
--webserver_interface=127.31.40.3
--webserver_port=0
--tserver_master_addrs=127.31.40.62:40205
--builtin_ntp_servers=127.31.40.20:32833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 15:01:23.396427 18509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:23.384891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:36305 (local address 127.31.40.62:40205)
0928 15:01:23.385093 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:23.385098 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:23.386680 (+ 1582us) server_negotiation.cc:407] Connection header received
0928 15:01:23.388612 (+ 1932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:23.388618 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:23.388713 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:23.388825 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:23.390094 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:23.391080 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:23.392951 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:23.393734 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:23.393881 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:23.395460 (+ 1579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:23.395469 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:23.395476 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:01:23.395501 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:23.395545 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:01:23.395562 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:23.395656 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:23.395915 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 15:01:23.396028 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 15:01:23.397449 18640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:23.385473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40205 (local address 127.31.40.2:36305)
0928 15:01:23.386518 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:23.386571 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:23.388319 (+ 1748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:23.389066 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:23.389074 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:23.389159 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:01:23.389915 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:23.389940 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:23.391244 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:23.391249 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:23.392804 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:23.392817 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:23.393992 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:23.393998 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:23.394302 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:23.395095 (+ 793us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:01:23.395098 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:01:23.395140 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:01:23.395223 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:01:23.395664 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:01:23.395672 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:01:23.395678 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 15:01:23.395840 (+ 162us) client_negotiation.cc:240] Negotiation successful
0928 15:01:23.396226 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":297,"threads_started":1}
I20260928 15:01:23.399060 18638 heartbeater.cc:347] Connected to a master server at 127.31.40.62:40205
I20260928 15:01:23.399427 18638 heartbeater.cc:464] Registering TS with master...
I20260928 15:01:23.400528 18638 heartbeater.cc:511] Master 127.31.40.62:40205 requested a full tablet report, sending...
I20260928 15:01:23.402241 18321 ts_manager.cc:194] Registered new tserver with Master: f314588e5b6445c7a4bc17c9af2db7cb (127.31.40.2:34177)
I20260928 15:01:23.403560 18321 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.40.2:36305
W20260928 15:01:23.731590 18641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:23.731959 18641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:23.732088 18641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:23.739488 18641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:23.739602 18641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:23.739711 18641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.3
I20260928 15:01:23.749185 18641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:32833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.40.3
--webserver_port=0
--tserver_master_addrs=127.31.40.62:40205
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.3
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:01:23.750546 18641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:23.752372 18641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:23.766299 18646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:23.766366 18649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:23.766336 18647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:23.767100 18641 server_base.cc:1034] running on GCE node
I20260928 15:01:23.767752 18641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:23.768929 18641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:23.770164 18641 hybrid_clock.cc:648] HybridClock initialized: now 1790607683770126 us; error 56 us; skew 500 ppm
I20260928 15:01:23.770515 18641 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:01:23.773077 18641 webserver.cc:460] Webserver started at http://127.31.40.3:35659/ using document root <none> and password file <none>
I20260928 15:01:23.773830 18641 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:23.773983 18641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:23.774324 18641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:23.777446 18641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/ts-2/data/instance:
uuid: "bf32b070f04c44bfa832e016c0976a9f"
format_stamp: "Formatted at 2026-09-28 15:01:23 on dist-test-slave-j968"
server_key: "f4e57453e75803fee959b590d45e4ec4"
server_key_iv: "7111b848733b01ed970f7bebd2aea027"
server_key_version: "encryptionkey@0"
I20260928 15:01:23.778263 18641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/ts-2/wal/instance:
uuid: "bf32b070f04c44bfa832e016c0976a9f"
format_stamp: "Formatted at 2026-09-28 15:01:23 on dist-test-slave-j968"
server_key: "f4e57453e75803fee959b590d45e4ec4"
server_key_iv: "7111b848733b01ed970f7bebd2aea027"
server_key_version: "encryptionkey@0"
I20260928 15:01:23.783758 18641 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 15:01:23.787518 18655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:23.788937 18641 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 15:01:23.789116 18641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/ts-2/wal
uuid: "bf32b070f04c44bfa832e016c0976a9f"
format_stamp: "Formatted at 2026-09-28 15:01:23 on dist-test-slave-j968"
server_key: "f4e57453e75803fee959b590d45e4ec4"
server_key_iv: "7111b848733b01ed970f7bebd2aea027"
server_key_version: "encryptionkey@0"
I20260928 15:01:23.789335 18641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:23.811432 18641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:23.812397 18641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:23.812690 18641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:23.813756 18641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:01:23.815302 18641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:01:23.815436 18641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:23.815575 18641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:01:23.815665 18641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:23.852329 18641 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.3:34103
I20260928 15:01:23.852432 18767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.3:34103 every 8 connection(s)
I20260928 15:01:23.853708 18641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
I20260928 15:01:23.860450 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 18641
I20260928 15:01:23.860622 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/ts-2/wal/instance
I20260928 15:01:23.861119 31904 external_mini_cluster.cc:1428] Setting key decf5e79cd7229d4c3739fbafe7464ee
I20260928 15:01:23.866799 18509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:23.856427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:47559 (local address 127.31.40.62:40205)
0928 15:01:23.856609 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:23.856614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:23.858015 (+ 1401us) server_negotiation.cc:407] Connection header received
0928 15:01:23.859680 (+ 1665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:23.859685 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:23.859759 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:23.859848 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:23.861118 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:23.862166 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:23.864417 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:23.864820 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:23.864925 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:23.866146 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:23.866151 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:23.866154 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:01:23.866171 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:23.866202 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:01:23.866214 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:23.866283 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:23.866495 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 15:01:23.866570 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 15:01:23.867782 18770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:23.856955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:40205 (local address 127.31.40.3:47559)
0928 15:01:23.857838 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:23.857902 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:23.859443 (+ 1541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:23.860170 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:23.860178 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:23.860269 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:01:23.860945 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:23.860962 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:23.862316 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:23.862325 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:23.864293 (+ 1968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:23.864305 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:23.865005 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:23.865010 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:23.865201 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:23.865850 (+ 649us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:01:23.865853 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:01:23.865894 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:01:23.865977 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:01:23.866296 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:01:23.866303 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:01:23.866308 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 15:01:23.866469 (+ 161us) client_negotiation.cc:240] Negotiation successful
0928 15:01:23.866757 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":256,"threads_started":1}
I20260928 15:01:23.869098 18768 heartbeater.cc:347] Connected to a master server at 127.31.40.62:40205
I20260928 15:01:23.869362 18768 heartbeater.cc:464] Registering TS with master...
I20260928 15:01:23.870043 18768 heartbeater.cc:511] Master 127.31.40.62:40205 requested a full tablet report, sending...
I20260928 15:01:23.871359 18321 ts_manager.cc:194] Registered new tserver with Master: bf32b070f04c44bfa832e016c0976a9f (127.31.40.3:34103)
I20260928 15:01:23.872241 18321 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.40.3:47559
I20260928 15:01:23.876705 31904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:01:23.887326 18509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:23.881359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52228 (local address 127.31.40.62:40205)
0928 15:01:23.881587 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:23.881595 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:23.882086 (+ 491us) server_negotiation.cc:407] Connection header received
0928 15:01:23.882201 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:23.882205 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:23.882278 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:23.882358 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:23.882784 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:23.883465 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:23.884976 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:23.885397 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:23.885489 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:23.886739 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:23.886744 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:23.886747 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:01:23.886763 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:23.886791 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:01:23.886802 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:23.886912 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:23.887071 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 15:01:23.887114 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260928 15:01:23.891144 18321 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52228:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:01:23.898241 18321 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:01:23.930650 18780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:23.910159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60558 (local address 127.31.40.2:34177)
0928 15:01:23.920341 (+ 10182us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:23.920352 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:23.920405 (+ 53us) server_negotiation.cc:407] Connection header received
0928 15:01:23.920535 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:23.920546 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:23.920867 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:23.922117 (+ 1250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:23.922435 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:23.924482 (+ 2047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:23.927629 (+ 3147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:23.929396 (+ 1767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:23.929572 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:23.929829 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:23.930139 (+ 310us) server_negotiation.cc:299] Negotiation successful
0928 15:01:23.930306 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9976,"thread_start_us":15881,"threads_started":1}
I20260928 15:01:23.930632 18781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:23.910108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.2:34177 (local address 127.0.0.1:60558)
0928 15:01:23.917749 (+ 7641us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:23.917815 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:23.918034 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:23.921154 (+ 3120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:23.921165 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:23.921220 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:23.921824 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:23.921842 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:23.924680 (+ 2838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:23.924688 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:23.927359 (+ 2671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:23.927382 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:23.929604 (+ 2222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:23.929610 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:23.929857 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:23.929927 (+ 70us) client_negotiation.cc:769] Sending connection context
0928 15:01:23.930070 (+ 143us) client_negotiation.cc:240] Negotiation successful
0928 15:01:23.930266 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7272,"thread_start_us":7043,"threads_started":1}
I20260928 15:01:23.934093 18782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:23.910257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:40577 (local address 127.0.0.1:56080)
0928 15:01:23.925981 (+ 15724us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:23.926001 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:23.926403 (+ 402us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:23.927329 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:23.927334 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:23.927362 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:23.927698 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:23.927707 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:23.929589 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:23.929596 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:23.931756 (+ 2160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:23.931770 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:23.933350 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:23.933357 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:23.933531 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:23.933561 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 15:01:23.933669 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 15:01:23.933787 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15548,"thread_start_us":15873,"threads_started":1}
I20260928 15:01:23.934248 18783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:23.910784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56080 (local address 127.31.40.1:40577)
0928 15:01:23.920018 (+ 9234us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:23.920030 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:23.926100 (+ 6070us) server_negotiation.cc:407] Connection header received
0928 15:01:23.926605 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:23.926615 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:23.926880 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:23.927043 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:23.927895 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:23.929387 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:23.932030 (+ 2643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:23.933058 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:23.933207 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:23.933469 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:23.933714 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 15:01:23.933882 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9002,"thread_start_us":8868,"threads_started":1}
I20260928 15:01:23.935189 18573 tablet_service.cc:1467] Processing CreateTablet for tablet b363e87437144629b0545a4734c4a8ed (DEFAULT_TABLE table=test-table [id=0cc27b427a514849818e41a66010e94d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:01:23.936601 18785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:23.922237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.3:34103 (local address 127.0.0.1:45430)
0928 15:01:23.926762 (+ 4525us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:23.926786 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:23.926958 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:23.928447 (+ 1489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:23.928457 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:23.928481 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:23.928927 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:23.928940 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:23.930927 (+ 1987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:23.930932 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:23.933073 (+ 2141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:23.933087 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:23.935886 (+ 2799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:23.935894 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:23.936075 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:23.936107 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 15:01:23.936215 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 15:01:23.936340 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4352,"thread_start_us":4216,"threads_started":1}
I20260928 15:01:23.936753 18784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:23.918353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45430 (local address 127.31.40.3:34103)
0928 15:01:23.927589 (+ 9236us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:23.927599 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:23.927631 (+ 32us) server_negotiation.cc:407] Connection header received
0928 15:01:23.927725 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:23.927733 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:23.927990 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:23.928182 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:23.929134 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:23.930721 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:23.933283 (+ 2562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:23.935643 (+ 2360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:23.935776 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:23.936081 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:23.936263 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 15:01:23.936426 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9054,"thread_start_us":8903,"threads_started":1}
I20260928 15:01:23.937052 18573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b363e87437144629b0545a4734c4a8ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:23.937879 18442 tablet_service.cc:1467] Processing CreateTablet for tablet b363e87437144629b0545a4734c4a8ed (DEFAULT_TABLE table=test-table [id=0cc27b427a514849818e41a66010e94d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:01:23.939391 18442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b363e87437144629b0545a4734c4a8ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:23.940526 18703 tablet_service.cc:1467] Processing CreateTablet for tablet b363e87437144629b0545a4734c4a8ed (DEFAULT_TABLE table=test-table [id=0cc27b427a514849818e41a66010e94d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:01:23.941922 18703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b363e87437144629b0545a4734c4a8ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:23.943593 18507 heartbeater.cc:503] Master 127.31.40.62:40205 was elected leader, sending a full tablet report...
I20260928 15:01:23.948828 18786 tablet_bootstrap.cc:492] T b363e87437144629b0545a4734c4a8ed P f314588e5b6445c7a4bc17c9af2db7cb: Bootstrap starting.
I20260928 15:01:23.952826 18786 tablet_bootstrap.cc:654] T b363e87437144629b0545a4734c4a8ed P f314588e5b6445c7a4bc17c9af2db7cb: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:23.954066 18787 tablet_bootstrap.cc:492] T b363e87437144629b0545a4734c4a8ed P 0ad302c7657f4607b1c4d4cf61aed260: Bootstrap starting.
I20260928 15:01:23.954058 18786 log.cc:824] T b363e87437144629b0545a4734c4a8ed P f314588e5b6445c7a4bc17c9af2db7cb: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:23.956302 18789 tablet_bootstrap.cc:492] T b363e87437144629b0545a4734c4a8ed P bf32b070f04c44bfa832e016c0976a9f: Bootstrap starting.
I20260928 15:01:23.958564 18787 tablet_bootstrap.cc:654] T b363e87437144629b0545a4734c4a8ed P 0ad302c7657f4607b1c4d4cf61aed260: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:23.959038 18789 tablet_bootstrap.cc:654] T b363e87437144629b0545a4734c4a8ed P bf32b070f04c44bfa832e016c0976a9f: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:23.960119 18787 log.cc:824] T b363e87437144629b0545a4734c4a8ed P 0ad302c7657f4607b1c4d4cf61aed260: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:23.960261 18789 log.cc:824] T b363e87437144629b0545a4734c4a8ed P bf32b070f04c44bfa832e016c0976a9f: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:23.964412 18786 tablet_bootstrap.cc:492] T b363e87437144629b0545a4734c4a8ed P f314588e5b6445c7a4bc17c9af2db7cb: No bootstrap required, opened a new log
I20260928 15:01:23.964779 18786 ts_tablet_manager.cc:1397] T b363e87437144629b0545a4734c4a8ed P f314588e5b6445c7a4bc17c9af2db7cb: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.004s
I20260928 15:01:23.966117 18789 tablet_bootstrap.cc:492] T b363e87437144629b0545a4734c4a8ed P bf32b070f04c44bfa832e016c0976a9f: No bootstrap required, opened a new log
I20260928 15:01:23.966394 18789 ts_tablet_manager.cc:1397] T b363e87437144629b0545a4734c4a8ed P bf32b070f04c44bfa832e016c0976a9f: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260928 15:01:23.970484 18787 tablet_bootstrap.cc:492] T b363e87437144629b0545a4734c4a8ed P 0ad302c7657f4607b1c4d4cf61aed260: No bootstrap required, opened a new log
I20260928 15:01:23.970753 18787 ts_tablet_manager.cc:1397] T b363e87437144629b0545a4734c4a8ed P 0ad302c7657f4607b1c4d4cf61aed260: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.010s
I20260928 15:01:23.973816 18789 raft_consensus.cc:348] T b363e87437144629b0545a4734c4a8ed P bf32b070f04c44bfa832e016c0976a9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f314588e5b6445c7a4bc17c9af2db7cb" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34177 } } peers { permanent_uuid: "0ad302c7657f4607b1c4d4cf61aed260" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 40577 } } peers { permanent_uuid: "bf32b070f04c44bfa832e016c0976a9f" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34103 } }
I20260928 15:01:23.974277 18789 raft_consensus.cc:374] T b363e87437144629b0545a4734c4a8ed P bf32b070f04c44bfa832e016c0976a9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:23.974390 18789 raft_consensus.cc:729] T b363e87437144629b0545a4734c4a8ed P bf32b070f04c44bfa832e016c0976a9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf32b070f04c44bfa832e016c0976a9f, State: Initialized, Role: FOLLOWER
I20260928 15:01:23.975220 18789 consensus_queue.cc:260] T b363e87437144629b0545a4734c4a8ed P bf32b070f04c44bfa832e016c0976a9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f314588e5b6445c7a4bc17c9af2db7cb" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34177 } } peers { permanent_uuid: "0ad302c7657f4607b1c4d4cf61aed260" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 40577 } } peers { permanent_uuid: "bf32b070f04c44bfa832e016c0976a9f" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34103 } }
I20260928 15:01:23.975682 18786 raft_consensus.cc:348] T b363e87437144629b0545a4734c4a8ed P f314588e5b6445c7a4bc17c9af2db7cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f314588e5b6445c7a4bc17c9af2db7cb" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34177 } } peers { permanent_uuid: "0ad302c7657f4607b1c4d4cf61aed260" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 40577 } } peers { permanent_uuid: "bf32b070f04c44bfa832e016c0976a9f" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34103 } }
I20260928 15:01:23.976171 18786 raft_consensus.cc:374] T b363e87437144629b0545a4734c4a8ed P f314588e5b6445c7a4bc17c9af2db7cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:23.976275 18786 raft_consensus.cc:729] T b363e87437144629b0545a4734c4a8ed P f314588e5b6445c7a4bc17c9af2db7cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f314588e5b6445c7a4bc17c9af2db7cb, State: Initialized, Role: FOLLOWER
I20260928 15:01:23.976496 18768 heartbeater.cc:503] Master 127.31.40.62:40205 was elected leader, sending a full tablet report...
I20260928 15:01:23.977097 18789 ts_tablet_manager.cc:1428] T b363e87437144629b0545a4734c4a8ed P bf32b070f04c44bfa832e016c0976a9f: Time spent starting tablet: real 0.011s user 0.009s sys 0.002s
I20260928 15:01:23.976984 18786 consensus_queue.cc:260] T b363e87437144629b0545a4734c4a8ed P f314588e5b6445c7a4bc17c9af2db7cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f314588e5b6445c7a4bc17c9af2db7cb" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34177 } } peers { permanent_uuid: "0ad302c7657f4607b1c4d4cf61aed260" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 40577 } } peers { permanent_uuid: "bf32b070f04c44bfa832e016c0976a9f" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34103 } }
I20260928 15:01:23.978103 18638 heartbeater.cc:503] Master 127.31.40.62:40205 was elected leader, sending a full tablet report...
I20260928 15:01:23.978878 18786 ts_tablet_manager.cc:1428] T b363e87437144629b0545a4734c4a8ed P f314588e5b6445c7a4bc17c9af2db7cb: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260928 15:01:23.981976 18787 raft_consensus.cc:348] T b363e87437144629b0545a4734c4a8ed P 0ad302c7657f4607b1c4d4cf61aed260 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f314588e5b6445c7a4bc17c9af2db7cb" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34177 } } peers { permanent_uuid: "0ad302c7657f4607b1c4d4cf61aed260" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 40577 } } peers { permanent_uuid: "bf32b070f04c44bfa832e016c0976a9f" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34103 } }
I20260928 15:01:23.982448 18787 raft_consensus.cc:374] T b363e87437144629b0545a4734c4a8ed P 0ad302c7657f4607b1c4d4cf61aed260 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:23.982570 18787 raft_consensus.cc:729] T b363e87437144629b0545a4734c4a8ed P 0ad302c7657f4607b1c4d4cf61aed260 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ad302c7657f4607b1c4d4cf61aed260, State: Initialized, Role: FOLLOWER
I20260928 15:01:23.983198 18787 consensus_queue.cc:260] T b363e87437144629b0545a4734c4a8ed P 0ad302c7657f4607b1c4d4cf61aed260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f314588e5b6445c7a4bc17c9af2db7cb" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34177 } } peers { permanent_uuid: "0ad302c7657f4607b1c4d4cf61aed260" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 40577 } } peers { permanent_uuid: "bf32b070f04c44bfa832e016c0976a9f" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34103 } }
I20260928 15:01:23.984853 18787 ts_tablet_manager.cc:1428] T b363e87437144629b0545a4734c4a8ed P 0ad302c7657f4607b1c4d4cf61aed260: Time spent starting tablet: real 0.014s user 0.009s sys 0.006s
I20260928 15:01:24.081605 18792 raft_consensus.cc:482] T b363e87437144629b0545a4734c4a8ed P bf32b070f04c44bfa832e016c0976a9f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:01:24.081987 18792 raft_consensus.cc:504] T b363e87437144629b0545a4734c4a8ed P bf32b070f04c44bfa832e016c0976a9f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f314588e5b6445c7a4bc17c9af2db7cb" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34177 } } peers { permanent_uuid: "0ad302c7657f4607b1c4d4cf61aed260" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 40577 } } peers { permanent_uuid: "bf32b070f04c44bfa832e016c0976a9f" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34103 } }
I20260928 15:01:24.083925 18792 leader_election.cc:288] T b363e87437144629b0545a4734c4a8ed P bf32b070f04c44bfa832e016c0976a9f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f314588e5b6445c7a4bc17c9af2db7cb (127.31.40.2:34177), 0ad302c7657f4607b1c4d4cf61aed260 (127.31.40.1:40577)
I20260928 15:01:24.090606 18783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:24.084374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:46099 (local address 127.31.40.1:40577)
0928 15:01:24.084582 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:24.084589 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:24.084788 (+ 199us) server_negotiation.cc:407] Connection header received
0928 15:01:24.085015 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:24.085021 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:24.085122 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:24.085247 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:24.085795 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:24.086783 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:24.088697 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:24.089645 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:24.089744 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:24.089821 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:24.090191 (+ 370us) server_negotiation.cc:299] Negotiation successful
0928 15:01:24.090358 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 15:01:24.090603 18770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:24.084192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:40577 (local address 127.31.40.3:46099)
0928 15:01:24.084647 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:24.084675 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:24.084846 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:24.085259 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:24.085263 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:24.085311 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:24.085562 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:24.085574 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:24.086932 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:24.086937 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:24.088535 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:24.088549 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:24.089798 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:24.089803 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:24.090018 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:24.090059 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 15:01:24.090149 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 15:01:24.090288 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304}
I20260928 15:01:24.090914 18780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:24.084405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:46359 (local address 127.31.40.2:34177)
0928 15:01:24.084604 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:24.084612 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:24.085324 (+ 712us) server_negotiation.cc:407] Connection header received
0928 15:01:24.085519 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:24.085525 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:24.085612 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:24.085706 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:24.086381 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:24.087200 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:24.089117 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:24.089951 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:24.090071 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:24.090176 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:24.090500 (+ 324us) server_negotiation.cc:299] Negotiation successful
0928 15:01:24.090639 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 15:01:24.090994 18795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:24.084323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.2:34177 (local address 127.31.40.3:46359)
0928 15:01:24.085202 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:24.085229 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:24.085363 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:24.085795 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:24.085800 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:24.085826 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:24.086209 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:24.086221 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:24.087352 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:24.087356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:24.088929 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:24.088945 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:24.090133 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:24.090142 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:24.090323 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:24.090355 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 15:01:24.090451 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 15:01:24.090568 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"mutex_wait_us":1,"thread_start_us":436,"threads_started":1}
I20260928 15:01:24.092058 18593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b363e87437144629b0545a4734c4a8ed" candidate_uuid: "bf32b070f04c44bfa832e016c0976a9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f314588e5b6445c7a4bc17c9af2db7cb" is_pre_election: true
I20260928 15:01:24.092060 18462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b363e87437144629b0545a4734c4a8ed" candidate_uuid: "bf32b070f04c44bfa832e016c0976a9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ad302c7657f4607b1c4d4cf61aed260" is_pre_election: true
I20260928 15:01:24.092674 18462 raft_consensus.cc:2454] T b363e87437144629b0545a4734c4a8ed P 0ad302c7657f4607b1c4d4cf61aed260 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf32b070f04c44bfa832e016c0976a9f in term 0.
I20260928 15:01:24.092674 18593 raft_consensus.cc:2454] T b363e87437144629b0545a4734c4a8ed P f314588e5b6445c7a4bc17c9af2db7cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf32b070f04c44bfa832e016c0976a9f in term 0.
I20260928 15:01:24.093412 18658 leader_election.cc:302] T b363e87437144629b0545a4734c4a8ed P bf32b070f04c44bfa832e016c0976a9f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf32b070f04c44bfa832e016c0976a9f, f314588e5b6445c7a4bc17c9af2db7cb; no voters:
I20260928 15:01:24.093866 18792 raft_consensus.cc:2781] T b363e87437144629b0545a4734c4a8ed P bf32b070f04c44bfa832e016c0976a9f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:01:24.094007 18792 raft_consensus.cc:482] T b363e87437144629b0545a4734c4a8ed P bf32b070f04c44bfa832e016c0976a9f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:01:24.094136 18792 raft_consensus.cc:3037] T b363e87437144629b0545a4734c4a8ed P bf32b070f04c44bfa832e016c0976a9f [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:24.096356 18792 raft_consensus.cc:504] T b363e87437144629b0545a4734c4a8ed P bf32b070f04c44bfa832e016c0976a9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f314588e5b6445c7a4bc17c9af2db7cb" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34177 } } peers { permanent_uuid: "0ad302c7657f4607b1c4d4cf61aed260" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 40577 } } peers { permanent_uuid: "bf32b070f04c44bfa832e016c0976a9f" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34103 } }
I20260928 15:01:24.096994 18792 leader_election.cc:288] T b363e87437144629b0545a4734c4a8ed P bf32b070f04c44bfa832e016c0976a9f [CANDIDATE]: Term 1 election: Requested vote from peers f314588e5b6445c7a4bc17c9af2db7cb (127.31.40.2:34177), 0ad302c7657f4607b1c4d4cf61aed260 (127.31.40.1:40577)
I20260928 15:01:24.097441 18593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b363e87437144629b0545a4734c4a8ed" candidate_uuid: "bf32b070f04c44bfa832e016c0976a9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f314588e5b6445c7a4bc17c9af2db7cb"
I20260928 15:01:24.097456 18462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b363e87437144629b0545a4734c4a8ed" candidate_uuid: "bf32b070f04c44bfa832e016c0976a9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ad302c7657f4607b1c4d4cf61aed260"
I20260928 15:01:24.097666 18593 raft_consensus.cc:3037] T b363e87437144629b0545a4734c4a8ed P f314588e5b6445c7a4bc17c9af2db7cb [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:24.097661 18462 raft_consensus.cc:3037] T b363e87437144629b0545a4734c4a8ed P 0ad302c7657f4607b1c4d4cf61aed260 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:24.100955 18462 raft_consensus.cc:2454] T b363e87437144629b0545a4734c4a8ed P 0ad302c7657f4607b1c4d4cf61aed260 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf32b070f04c44bfa832e016c0976a9f in term 1.
I20260928 15:01:24.100956 18593 raft_consensus.cc:2454] T b363e87437144629b0545a4734c4a8ed P f314588e5b6445c7a4bc17c9af2db7cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf32b070f04c44bfa832e016c0976a9f in term 1.
I20260928 15:01:24.101370 18658 leader_election.cc:302] T b363e87437144629b0545a4734c4a8ed P bf32b070f04c44bfa832e016c0976a9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf32b070f04c44bfa832e016c0976a9f, f314588e5b6445c7a4bc17c9af2db7cb; no voters:
I20260928 15:01:24.101691 18792 raft_consensus.cc:2781] T b363e87437144629b0545a4734c4a8ed P bf32b070f04c44bfa832e016c0976a9f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:01:24.102203 18792 raft_consensus.cc:686] T b363e87437144629b0545a4734c4a8ed P bf32b070f04c44bfa832e016c0976a9f [term 1 LEADER]: Becoming Leader. State: Replica: bf32b070f04c44bfa832e016c0976a9f, State: Running, Role: LEADER
I20260928 15:01:24.103072 18792 consensus_queue.cc:237] T b363e87437144629b0545a4734c4a8ed P bf32b070f04c44bfa832e016c0976a9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f314588e5b6445c7a4bc17c9af2db7cb" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34177 } } peers { permanent_uuid: "0ad302c7657f4607b1c4d4cf61aed260" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 40577 } } peers { permanent_uuid: "bf32b070f04c44bfa832e016c0976a9f" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34103 } }
W20260928 15:01:24.105465 18769 tablet.cc:2369] T b363e87437144629b0545a4734c4a8ed P bf32b070f04c44bfa832e016c0976a9f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:01:24.108065 18321 catalog_manager.cc:5495] T b363e87437144629b0545a4734c4a8ed P bf32b070f04c44bfa832e016c0976a9f reported cstate change: term changed from 0 to 1, leader changed from <none> to bf32b070f04c44bfa832e016c0976a9f (127.31.40.3). New cstate: current_term: 1 leader_uuid: "bf32b070f04c44bfa832e016c0976a9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f314588e5b6445c7a4bc17c9af2db7cb" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ad302c7657f4607b1c4d4cf61aed260" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf32b070f04c44bfa832e016c0976a9f" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 34103 } health_report { overall_health: HEALTHY } } }
W20260928 15:01:24.134431 18639 tablet.cc:2369] T b363e87437144629b0545a4734c4a8ed P f314588e5b6445c7a4bc17c9af2db7cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:01:24.162051 18784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:24.156577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45446 (local address 127.31.40.3:34103)
0928 15:01:24.156824 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:24.156829 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:24.156856 (+ 27us) server_negotiation.cc:407] Connection header received
0928 15:01:24.156940 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:24.156943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:24.157027 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:24.157151 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:01:24.157488 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:24.158283 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:24.159626 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:24.160102 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:24.160198 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:24.160741 (+ 543us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:01:24.161389 (+ 648us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:01:24.161596 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:24.161691 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 15:01:24.161765 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 15:01:24.170387 18462 raft_consensus.cc:1261] T b363e87437144629b0545a4734c4a8ed P 0ad302c7657f4607b1c4d4cf61aed260 [term 1 FOLLOWER]: Refusing update from remote peer bf32b070f04c44bfa832e016c0976a9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:01:24.170506 18593 raft_consensus.cc:1261] T b363e87437144629b0545a4734c4a8ed P f314588e5b6445c7a4bc17c9af2db7cb [term 1 FOLLOWER]: Refusing update from remote peer bf32b070f04c44bfa832e016c0976a9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260928 15:01:24.171484 18508 tablet.cc:2369] T b363e87437144629b0545a4734c4a8ed P 0ad302c7657f4607b1c4d4cf61aed260: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:01:24.171491 18796 consensus_queue.cc:1035] T b363e87437144629b0545a4734c4a8ed P bf32b070f04c44bfa832e016c0976a9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ad302c7657f4607b1c4d4cf61aed260" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:01:24.171914 18792 consensus_queue.cc:1035] T b363e87437144629b0545a4734c4a8ed P bf32b070f04c44bfa832e016c0976a9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f314588e5b6445c7a4bc17c9af2db7cb" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:01:24.189214 18799 mvcc.cc:205] Tried to move back new op lower bound from 7334329074351693824 to 7334329074090848256. Current Snapshot: MvccSnapshot[applied={T|T < 7334329074351693824}]
I20260928 15:01:24.202728 18320 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:01:24.203102 18320 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:01:24.207134 18320 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 17d16212cd294f0fbdf5904bf61d6c8a: Sending DeleteTablet for 3 replicas of tablet b363e87437144629b0545a4734c4a8ed
I20260928 15:01:24.208534 18442 tablet_service.cc:1514] Processing DeleteTablet for tablet b363e87437144629b0545a4734c4a8ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:01:24 UTC) from {username='slave'} at 127.0.0.1:56080
I20260928 15:01:24.208741 18703 tablet_service.cc:1514] Processing DeleteTablet for tablet b363e87437144629b0545a4734c4a8ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:01:24 UTC) from {username='slave'} at 127.0.0.1:45430
I20260928 15:01:24.208714 18573 tablet_service.cc:1514] Processing DeleteTablet for tablet b363e87437144629b0545a4734c4a8ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:01:24 UTC) from {username='slave'} at 127.0.0.1:60558
I20260928 15:01:24.210634 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 18380
I20260928 15:01:24.219036 18810 tablet_replica.cc:330] T b363e87437144629b0545a4734c4a8ed P f314588e5b6445c7a4bc17c9af2db7cb: stopping tablet replica
I20260928 15:01:24.219688 18810 raft_consensus.cc:2229] T b363e87437144629b0545a4734c4a8ed P f314588e5b6445c7a4bc17c9af2db7cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:01:24.220431 18810 raft_consensus.cc:2258] T b363e87437144629b0545a4734c4a8ed P f314588e5b6445c7a4bc17c9af2db7cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:01:24.221532 18809 tablet_replica.cc:330] T b363e87437144629b0545a4734c4a8ed P 0ad302c7657f4607b1c4d4cf61aed260: stopping tablet replica
I20260928 15:01:24.222112 18809 raft_consensus.cc:2229] T b363e87437144629b0545a4734c4a8ed P 0ad302c7657f4607b1c4d4cf61aed260 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:01:24.222738 18809 raft_consensus.cc:2258] T b363e87437144629b0545a4734c4a8ed P 0ad302c7657f4607b1c4d4cf61aed260 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:01:24.225037 18810 ts_tablet_manager.cc:1905] T b363e87437144629b0545a4734c4a8ed P f314588e5b6445c7a4bc17c9af2db7cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:01:24.226503 18809 ts_tablet_manager.cc:1905] T b363e87437144629b0545a4734c4a8ed P 0ad302c7657f4607b1c4d4cf61aed260: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:01:24.230690 18811 tablet_replica.cc:330] T b363e87437144629b0545a4734c4a8ed P bf32b070f04c44bfa832e016c0976a9f: stopping tablet replica
I20260928 15:01:24.232764 18810 ts_tablet_manager.cc:1918] T b363e87437144629b0545a4734c4a8ed P f314588e5b6445c7a4bc17c9af2db7cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:01:24.232946 18810 log.cc:1196] T b363e87437144629b0545a4734c4a8ed P f314588e5b6445c7a4bc17c9af2db7cb: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/ts-1/wal/wals/b363e87437144629b0545a4734c4a8ed
I20260928 15:01:24.233520 18810 ts_tablet_manager.cc:1939] T b363e87437144629b0545a4734c4a8ed P f314588e5b6445c7a4bc17c9af2db7cb: Deleting consensus metadata
I20260928 15:01:24.233680 18783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:24.212292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56082 (local address 127.31.40.1:40577)
0928 15:01:24.212494 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:24.212501 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:24.224379 (+ 11878us) server_negotiation.cc:407] Connection header received
0928 15:01:24.224638 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:24.224644 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:24.224752 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:24.224869 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:24.225596 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:24.226701 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:24.228765 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:24.229409 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:24.229507 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:24.231699 (+ 2192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:24.231720 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:24.231738 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:01:24.232608 (+ 870us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:24.232684 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:01:24.232715 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:24.232828 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:24.233130 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 15:01:24.233261 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 15:01:24.235041 18502 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:01:24.235538 18307 catalog_manager.cc:4841] TS f314588e5b6445c7a4bc17c9af2db7cb (127.31.40.2:34177): tablet b363e87437144629b0545a4734c4a8ed (table test-table [id=0cc27b427a514849818e41a66010e94d]) successfully deleted
I20260928 15:01:24.231176 18811 raft_consensus.cc:2229] T b363e87437144629b0545a4734c4a8ed P bf32b070f04c44bfa832e016c0976a9f [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:01:24.236690 18811 raft_consensus.cc:2258] T b363e87437144629b0545a4734c4a8ed P bf32b070f04c44bfa832e016c0976a9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:01:24.238897 18811 ts_tablet_manager.cc:1905] T b363e87437144629b0545a4734c4a8ed P bf32b070f04c44bfa832e016c0976a9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:01:24.244330 18811 ts_tablet_manager.cc:1918] T b363e87437144629b0545a4734c4a8ed P bf32b070f04c44bfa832e016c0976a9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:01:24.244518 18811 log.cc:1196] T b363e87437144629b0545a4734c4a8ed P bf32b070f04c44bfa832e016c0976a9f: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/ts-2/wal/wals/b363e87437144629b0545a4734c4a8ed
I20260928 15:01:24.244988 18811 ts_tablet_manager.cc:1939] T b363e87437144629b0545a4734c4a8ed P bf32b070f04c44bfa832e016c0976a9f: Deleting consensus metadata
I20260928 15:01:24.246510 18307 catalog_manager.cc:4841] TS bf32b070f04c44bfa832e016c0976a9f (127.31.40.3:34103): tablet b363e87437144629b0545a4734c4a8ed (table test-table [id=0cc27b427a514849818e41a66010e94d]) successfully deleted
I20260928 15:01:24.388865 18809 ts_tablet_manager.cc:1918] T b363e87437144629b0545a4734c4a8ed P 0ad302c7657f4607b1c4d4cf61aed260: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:01:24.389125 18809 log.cc:1196] T b363e87437144629b0545a4734c4a8ed P 0ad302c7657f4607b1c4d4cf61aed260: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607365362981-31904-0/minicluster-data/ts-0/wal/wals/b363e87437144629b0545a4734c4a8ed
I20260928 15:01:24.389590 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 18380
I20260928 15:01:24.389756 18809 ts_tablet_manager.cc:1939] T b363e87437144629b0545a4734c4a8ed P 0ad302c7657f4607b1c4d4cf61aed260: Deleting consensus metadata
I20260928 15:01:24.409308 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 18511
W20260928 15:01:24.409235 18305 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.40.1:40577 (error 108)
I20260928 15:01:24.413019 18785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:24.412447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:40577 (local address 127.0.0.1:56098)
0928 15:01:24.412671 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:24.412832 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.1:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260928 15:01:24.413321 18305 catalog_manager.cc:4584] TS 0ad302c7657f4607b1c4d4cf61aed260 (127.31.40.1:40577): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b363e87437144629b0545a4734c4a8ed: Network error: Client connection negotiation failed: client connection to 127.31.40.1:40577: connect: Connection refused (error 111)
I20260928 15:01:24.417407 18780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:24.410108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60568 (local address 127.31.40.2:34177)
0928 15:01:24.410322 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:24.410332 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:24.410394 (+ 62us) server_negotiation.cc:407] Connection header received
0928 15:01:24.410636 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:24.410645 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:24.410808 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:24.410952 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:24.411784 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:24.412810 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:24.414275 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:24.414669 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:24.414807 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:24.415996 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:24.416005 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:24.416021 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:01:24.416613 (+ 592us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:24.416667 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:01:24.416686 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:24.416817 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:24.416952 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 15:01:24.417067 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 15:01:24.418140 18633 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:01:24.467044 18785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:24.466454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:40577 (local address 127.0.0.1:56100)
0928 15:01:24.466733 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:24.466857 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.1:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260928 15:01:24.468258 18785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:24.467836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:40577 (local address 127.0.0.1:56102)
0928 15:01:24.468024 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:24.468111 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.1:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 15:01:24.508948 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 18511
I20260928 15:01:24.527778 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 18641
I20260928 15:01:24.528935 18785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:24.528432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:40577 (local address 127.0.0.1:56108)
0928 15:01:24.528648 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:24.528729 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.1:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 15:01:24.530093 18785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:24.529691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:40577 (local address 127.0.0.1:56116)
0928 15:01:24.529873 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:24.529974 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.1:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 15:01:24.536163 18784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:24.528946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45452 (local address 127.31.40.3:34103)
0928 15:01:24.529121 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:24.529127 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:24.529148 (+ 21us) server_negotiation.cc:407] Connection header received
0928 15:01:24.529298 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:24.529303 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:24.529403 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:24.529515 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:24.530014 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:24.531229 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:24.532573 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:24.533234 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:24.533388 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:24.534486 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:24.534498 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:24.534517 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:01:24.535322 (+ 805us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:24.535396 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:01:24.535423 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:24.535545 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:24.535651 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 15:01:24.535748 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 15:01:24.536866 18763 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:01:24.610872 18785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:24.610355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:40577 (local address 127.0.0.1:56118)
0928 15:01:24.610593 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:24.610680 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.1:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260928 15:01:24.612113 18785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:24.611645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:40577 (local address 127.0.0.1:56122)
0928 15:01:24.611925 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:24.612007 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.1:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260928 15:01:24.661408 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 18641
I20260928 15:01:24.683218 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 18290
I20260928 15:01:24.684365 18351 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:01:24.797626 18785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:24.796855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:40577 (local address 127.0.0.1:56136)
0928 15:01:24.797150 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:24.797269 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.1:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260928 15:01:24.797657 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 18290
2026-09-28T15:01:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3003 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:01:24 dist-test-slave-j968 krb5kdc[18820](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:01:24 dist-test-slave-j968 krb5kdc[18820](info): set up 2 sockets
Sep 28 15:01:24 dist-test-slave-j968 krb5kdc[18820](info): commencing operation
krb5kdc: starting...
W20260928 15:01:25.949215 31904 mini_kdc.cc:121] Time spent starting KDC: real 1.108s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Sep 28 15:01:26 dist-test-slave-j968 krb5kdc[18820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:01:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:01:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab.
Entry for principal kudu/127.31.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab.
I20260928 15:01:26.223515 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:42437
--webserver_interface=127.31.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:43107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.40.62:42437
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:01:26.573758 18833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:26.574095 18833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:26.574271 18833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:26.581653 18833 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:01:26.581790 18833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:26.581839 18833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:26.581966 18833 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:01:26.582053 18833 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:01:26.592381 18833 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:43107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.40.62:42437
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:42437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:01:26.593855 18833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:26.595208 18833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:26.608599 18838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:26.608943 18841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:26.609295 18839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:26.609808 18833 server_base.cc:1034] running on GCE node
I20260928 15:01:26.610608 18833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:26.612293 18833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:26.613510 18833 hybrid_clock.cc:648] HybridClock initialized: now 1790607686613464 us; error 30 us; skew 500 ppm
I20260928 15:01:26.613833 18833 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:01:26 dist-test-slave-j968 krb5kdc[18820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607686, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:01:26.616983 18833 init.cc:377] Logged in from keytab as kudu/127.31.40.62@KRBTEST.COM (short username kudu)
I20260928 15:01:26.618557 18833 webserver.cc:460] Webserver started at http://127.31.40.62:40919/ using document root <none> and password file <none>
I20260928 15:01:26.619256 18833 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:26.619398 18833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:26.619740 18833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:26.622786 18833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/master-0/data/instance:
uuid: "28647ebd484646b5ac7c03f702e24a18"
format_stamp: "Formatted at 2026-09-28 15:01:26 on dist-test-slave-j968"
server_key: "0abc83b6523e9359757fbde6b2679c82"
server_key_iv: "48f95a0aa8d7c9d65581919d6e95ffd5"
server_key_version: "encryptionkey@0"
I20260928 15:01:26.623554 18833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/master-0/wal/instance:
uuid: "28647ebd484646b5ac7c03f702e24a18"
format_stamp: "Formatted at 2026-09-28 15:01:26 on dist-test-slave-j968"
server_key: "0abc83b6523e9359757fbde6b2679c82"
server_key_iv: "48f95a0aa8d7c9d65581919d6e95ffd5"
server_key_version: "encryptionkey@0"
I20260928 15:01:26.628873 18833 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 15:01:26.632351 18848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:26.633675 18833 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 15:01:26.633862 18833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/master-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/master-0/wal
uuid: "28647ebd484646b5ac7c03f702e24a18"
format_stamp: "Formatted at 2026-09-28 15:01:26 on dist-test-slave-j968"
server_key: "0abc83b6523e9359757fbde6b2679c82"
server_key_iv: "48f95a0aa8d7c9d65581919d6e95ffd5"
server_key_version: "encryptionkey@0"
I20260928 15:01:26.634058 18833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:26.650978 18833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:26.655126 18833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:26.655454 18833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:26.675751 18833 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.62:42437
I20260928 15:01:26.675782 18899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.62:42437 every 8 connection(s)
I20260928 15:01:26.677237 18833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
I20260928 15:01:26.680908 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 18833
I20260928 15:01:26.681103 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/master-0/wal/instance
I20260928 15:01:26.681288 18900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:26.681648 31904 external_mini_cluster.cc:1428] Setting key 2096a99c7814b9735f5597cc984db6a8
I20260928 15:01:26.696938 18900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28647ebd484646b5ac7c03f702e24a18: Bootstrap starting.
Sep 28 15:01:26 dist-test-slave-j968 krb5kdc[18820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 15:01:26.701084 18900 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28647ebd484646b5ac7c03f702e24a18: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:26.702509 18900 log.cc:824] T 00000000000000000000000000000000 P 28647ebd484646b5ac7c03f702e24a18: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:26.705663 18900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28647ebd484646b5ac7c03f702e24a18: No bootstrap required, opened a new log
I20260928 15:01:26.707298 18903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:26.683580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57790 (local address 127.31.40.62:42437)
0928 15:01:26.684536 (+ 956us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:26.684552 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:26.684605 (+ 53us) server_negotiation.cc:407] Connection header received
0928 15:01:26.686537 (+ 1932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:26.686567 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:26.687116 (+ 549us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:26.687608 (+ 492us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:26.689089 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:26.691078 (+ 1989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:26.693022 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:26.694160 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:26.694391 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:26.698606 (+ 4215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:26.698619 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:26.698642 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:26.698697 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:26.702800 (+ 4103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:26.703761 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:26.703767 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:26.703773 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:26.704047 (+ 274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:26.704490 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:26.704495 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:26.704498 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:26.705034 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:26.705375 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:26.705830 (+ 455us) server_negotiation.cc:299] Negotiation successful
0928 15:01:26.706294 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":414,"threads_started":1}
I20260928 15:01:26.714476 18900 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28647ebd484646b5ac7c03f702e24a18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28647ebd484646b5ac7c03f702e24a18" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 42437 } }
I20260928 15:01:26.715035 18900 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28647ebd484646b5ac7c03f702e24a18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:26.715152 18900 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28647ebd484646b5ac7c03f702e24a18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28647ebd484646b5ac7c03f702e24a18, State: Initialized, Role: FOLLOWER
I20260928 15:01:26.715678 18900 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28647ebd484646b5ac7c03f702e24a18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28647ebd484646b5ac7c03f702e24a18" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 42437 } }
I20260928 15:01:26.715961 18900 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28647ebd484646b5ac7c03f702e24a18 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:01:26.716071 18900 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28647ebd484646b5ac7c03f702e24a18 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:01:26.716194 18900 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28647ebd484646b5ac7c03f702e24a18 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:26.718412 18900 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28647ebd484646b5ac7c03f702e24a18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28647ebd484646b5ac7c03f702e24a18" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 42437 } }
I20260928 15:01:26.719028 18900 leader_election.cc:302] T 00000000000000000000000000000000 P 28647ebd484646b5ac7c03f702e24a18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28647ebd484646b5ac7c03f702e24a18; no voters:
I20260928 15:01:26.719726 18900 leader_election.cc:288] T 00000000000000000000000000000000 P 28647ebd484646b5ac7c03f702e24a18 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:01:26.719956 18905 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28647ebd484646b5ac7c03f702e24a18 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:01:26.720711 18905 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28647ebd484646b5ac7c03f702e24a18 [term 1 LEADER]: Becoming Leader. State: Replica: 28647ebd484646b5ac7c03f702e24a18, State: Running, Role: LEADER
I20260928 15:01:26.721355 18905 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28647ebd484646b5ac7c03f702e24a18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28647ebd484646b5ac7c03f702e24a18" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 42437 } }
I20260928 15:01:26.722016 18900 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28647ebd484646b5ac7c03f702e24a18 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:01:26.724653 18907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28647ebd484646b5ac7c03f702e24a18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28647ebd484646b5ac7c03f702e24a18. Latest consensus state: current_term: 1 leader_uuid: "28647ebd484646b5ac7c03f702e24a18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28647ebd484646b5ac7c03f702e24a18" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 42437 } } }
I20260928 15:01:26.724581 18906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28647ebd484646b5ac7c03f702e24a18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28647ebd484646b5ac7c03f702e24a18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28647ebd484646b5ac7c03f702e24a18" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 42437 } } }
I20260928 15:01:26.725095 18907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28647ebd484646b5ac7c03f702e24a18 [sys.catalog]: This master's current role is: LEADER
I20260928 15:01:26.725098 18906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28647ebd484646b5ac7c03f702e24a18 [sys.catalog]: This master's current role is: LEADER
I20260928 15:01:26.729128 18914 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:01:26.734510 18914 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:01:26.744906 18914 catalog_manager.cc:1347] Generated new cluster ID: d64d1a7366fd4e6fa42fb915c16fc862
I20260928 15:01:26.745046 18914 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:01:26.773473 18914 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:01:26.774955 18914 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:01:26.784619 18914 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28647ebd484646b5ac7c03f702e24a18: Generated new TSK 0
I20260928 15:01:26.785602 18914 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.40.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab.
Entry for principal kudu/127.31.40.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab.
I20260928 15:01:26.859654 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.1:0
--local_ip_for_outbound_sockets=127.31.40.1
--webserver_interface=127.31.40.1
--webserver_port=0
--tserver_master_addrs=127.31.40.62:42437
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab
--principal=kudu/127.31.40.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:43107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:01:27.210649 18926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:27.210966 18926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:27.211122 18926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:27.219077 18926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:27.219213 18926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:27.219391 18926 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.1
I20260928 15:01:27.230613 18926 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:43107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.1.keytab
--principal=kudu/127.31.40.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.40.1
--webserver_port=0
--tserver_master_addrs=127.31.40.62:42437
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.1
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:01:27.232159 18926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:27.233542 18926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:27.248351 18932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:27.248391 18934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:27.248588 18931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:27.249150 18926 server_base.cc:1034] running on GCE node
I20260928 15:01:27.249825 18926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:27.250880 18926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:27.252113 18926 hybrid_clock.cc:648] HybridClock initialized: now 1790607687252045 us; error 55 us; skew 500 ppm
I20260928 15:01:27.252478 18926 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:01:27 dist-test-slave-j968 krb5kdc[18820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607687, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:01:27.256127 18926 init.cc:377] Logged in from keytab as kudu/127.31.40.1@KRBTEST.COM (short username kudu)
I20260928 15:01:27.257788 18926 webserver.cc:460] Webserver started at http://127.31.40.1:33467/ using document root <none> and password file <none>
I20260928 15:01:27.258571 18926 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:27.258737 18926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:27.259189 18926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:27.262506 18926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/ts-0/data/instance:
uuid: "a43673a7ceb443e8895168abcf255ea9"
format_stamp: "Formatted at 2026-09-28 15:01:27 on dist-test-slave-j968"
server_key: "0b3a4fa08891165a029fda3858574099"
server_key_iv: "24c4674b2b8f14845b9ac749e4c87de3"
server_key_version: "encryptionkey@0"
I20260928 15:01:27.263356 18926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/ts-0/wal/instance:
uuid: "a43673a7ceb443e8895168abcf255ea9"
format_stamp: "Formatted at 2026-09-28 15:01:27 on dist-test-slave-j968"
server_key: "0b3a4fa08891165a029fda3858574099"
server_key_iv: "24c4674b2b8f14845b9ac749e4c87de3"
server_key_version: "encryptionkey@0"
I20260928 15:01:27.269224 18926 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 15:01:27.273444 18941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:27.275089 18926 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:01:27.275291 18926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/ts-0/wal
uuid: "a43673a7ceb443e8895168abcf255ea9"
format_stamp: "Formatted at 2026-09-28 15:01:27 on dist-test-slave-j968"
server_key: "0b3a4fa08891165a029fda3858574099"
server_key_iv: "24c4674b2b8f14845b9ac749e4c87de3"
server_key_version: "encryptionkey@0"
I20260928 15:01:27.275524 18926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:27.303488 18926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:27.311105 18926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:27.311508 18926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:27.313035 18926 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:01:27.315164 18926 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:01:27.315282 18926 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:27.315459 18926 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:01:27.315541 18926 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:27.356442 18926 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.1:43049
I20260928 15:01:27.356485 19053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.1:43049 every 8 connection(s)
I20260928 15:01:27.357928 18926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
I20260928 15:01:27.360679 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 18926
I20260928 15:01:27.360982 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/ts-0/wal/instance
I20260928 15:01:27.361613 31904 external_mini_cluster.cc:1428] Setting key 2110658aa2bb3c7028b5f012727d6ab3
Sep 28 15:01:27 dist-test-slave-j968 krb5kdc[18820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607687, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.1@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 15:01:27.384831 19056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:27.361035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:53507 (local address 127.31.40.62:42437)
0928 15:01:27.361571 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:27.361577 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:27.362802 (+ 1225us) server_negotiation.cc:407] Connection header received
0928 15:01:27.365424 (+ 2622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:27.365428 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:27.365510 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:27.365607 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:27.367983 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:27.368816 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:27.371448 (+ 2632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:27.371968 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:27.372088 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:27.376535 (+ 4447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:27.376545 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:27.376555 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:27.376612 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:27.380770 (+ 4158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:27.381667 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:27.381677 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:27.381683 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:27.381883 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:27.382409 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:27.382418 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:27.382425 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:27.382871 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:27.383038 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:27.383983 (+ 945us) server_negotiation.cc:299] Negotiation successful
0928 15:01:27.384285 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":273,"threads_started":1}
I20260928 15:01:27.385687 19057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:27.361748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:42437 (local address 127.31.40.1:53507)
0928 15:01:27.362611 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:27.362666 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:27.365077 (+ 2411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:27.365922 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:27.365938 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:27.366780 (+ 842us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:01:27.367671 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:27.367702 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:27.369007 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:27.369015 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:27.371272 (+ 2257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:27.371290 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:27.372225 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:27.372241 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:27.372594 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:27.373402 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:01:27.373444 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:01:27.376230 (+ 2786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:01:27.380949 (+ 4719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:01:27.380957 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:01:27.380974 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:01:27.381508 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:01:27.382034 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:01:27.382037 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:01:27.382040 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:01:27.382245 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:01:27.383063 (+ 818us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:01:27.383070 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:01:27.383496 (+ 426us) client_negotiation.cc:769] Sending connection context
0928 15:01:27.383787 (+ 291us) client_negotiation.cc:240] Negotiation successful
0928 15:01:27.384362 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":316,"threads_started":1}
I20260928 15:01:27.387657 19054 heartbeater.cc:347] Connected to a master server at 127.31.40.62:42437
I20260928 15:01:27.388084 19054 heartbeater.cc:464] Registering TS with master...
I20260928 15:01:27.389212 19054 heartbeater.cc:511] Master 127.31.40.62:42437 requested a full tablet report, sending...
I20260928 15:01:27.392299 18865 ts_manager.cc:194] Registered new tserver with Master: a43673a7ceb443e8895168abcf255ea9 (127.31.40.1:43049)
WARNING: no policy specified for kudu/127.31.40.2@KRBTEST.COM; defaulting to no policy
I20260928 15:01:27.394080 18865 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.1@KRBTEST.COM'} at 127.31.40.1:53507
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab.
Entry for principal kudu/127.31.40.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab.
I20260928 15:01:27.428612 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.2:0
--local_ip_for_outbound_sockets=127.31.40.2
--webserver_interface=127.31.40.2
--webserver_port=0
--tserver_master_addrs=127.31.40.62:42437
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab
--principal=kudu/127.31.40.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:43107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:01:27.779270 19060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:27.779666 19060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:27.779847 19060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:27.787653 19060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:27.788002 19060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:27.788218 19060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.2
I20260928 15:01:27.798641 19060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:43107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.2.keytab
--principal=kudu/127.31.40.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.40.2
--webserver_port=0
--tserver_master_addrs=127.31.40.62:42437
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.2
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:01:27.800163 19060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:27.801672 19060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:27.817037 19065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:27.816967 19066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:27.816949 19068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:27.817941 19060 server_base.cc:1034] running on GCE node
I20260928 15:01:27.818634 19060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:27.819690 19060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:27.820976 19060 hybrid_clock.cc:648] HybridClock initialized: now 1790607687820907 us; error 61 us; skew 500 ppm
I20260928 15:01:27.821321 19060 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:01:27 dist-test-slave-j968 krb5kdc[18820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607687, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:01:27.825027 19060 init.cc:377] Logged in from keytab as kudu/127.31.40.2@KRBTEST.COM (short username kudu)
I20260928 15:01:27.827189 19060 webserver.cc:460] Webserver started at http://127.31.40.2:43223/ using document root <none> and password file <none>
I20260928 15:01:27.828104 19060 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:27.828250 19060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:27.828615 19060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:27.831983 19060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/ts-1/data/instance:
uuid: "32855210707a418495e2e2e139bf422e"
format_stamp: "Formatted at 2026-09-28 15:01:27 on dist-test-slave-j968"
server_key: "6110408768cacd1a932deb5d7d9d205a"
server_key_iv: "a6e5ee4ca0186c1bd36f821af1dec3f2"
server_key_version: "encryptionkey@0"
I20260928 15:01:27.832918 19060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/ts-1/wal/instance:
uuid: "32855210707a418495e2e2e139bf422e"
format_stamp: "Formatted at 2026-09-28 15:01:27 on dist-test-slave-j968"
server_key: "6110408768cacd1a932deb5d7d9d205a"
server_key_iv: "a6e5ee4ca0186c1bd36f821af1dec3f2"
server_key_version: "encryptionkey@0"
I20260928 15:01:27.839040 19060 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260928 15:01:27.843200 19075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:27.844918 19060 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 15:01:27.845135 19060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/ts-1/wal
uuid: "32855210707a418495e2e2e139bf422e"
format_stamp: "Formatted at 2026-09-28 15:01:27 on dist-test-slave-j968"
server_key: "6110408768cacd1a932deb5d7d9d205a"
server_key_iv: "a6e5ee4ca0186c1bd36f821af1dec3f2"
server_key_version: "encryptionkey@0"
I20260928 15:01:27.845358 19060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:27.867072 19060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:27.871505 19060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:27.871810 19060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:27.872884 19060 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:01:27.874342 19060 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:01:27.874456 19060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:27.874608 19060 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:01:27.874678 19060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:27.910027 19060 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.2:34419
I20260928 15:01:27.910069 19187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.2:34419 every 8 connection(s)
I20260928 15:01:27.911404 19060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
I20260928 15:01:27.917549 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 19060
I20260928 15:01:27.917825 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/ts-1/wal/instance
I20260928 15:01:27.918322 31904 external_mini_cluster.cc:1428] Setting key 4b3a6aad42e0e730b907c17757b70a70
Sep 28 15:01:27 dist-test-slave-j968 krb5kdc[18820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607687, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.2@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 15:01:27.936894 19190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:27.914329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:44063 (local address 127.31.40.62:42437)
0928 15:01:27.914834 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:27.914840 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:27.916099 (+ 1259us) server_negotiation.cc:407] Connection header received
0928 15:01:27.918205 (+ 2106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:27.918212 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:27.918323 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:27.918462 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:27.920555 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:27.921736 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:27.924107 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:27.924893 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:27.925056 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:27.929555 (+ 4499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:27.929564 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:27.929571 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:27.929632 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:27.932838 (+ 3206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:27.934057 (+ 1219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:27.934063 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:27.934065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:27.934243 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:27.934830 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:27.934834 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:27.934836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:27.935132 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:27.935302 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:27.936317 (+ 1015us) server_negotiation.cc:299] Negotiation successful
0928 15:01:27.936528 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":254,"threads_started":1}
I20260928 15:01:27.938347 19191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:27.914965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:42437 (local address 127.31.40.2:44063)
0928 15:01:27.915751 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:27.915963 (+ 212us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:27.917839 (+ 1876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:27.918766 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:27.918775 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:27.919528 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:01:27.920299 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:27.920320 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:27.921936 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:27.921940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:27.923901 (+ 1961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:27.923925 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:27.925168 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:27.925176 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:27.925558 (+ 382us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:27.926366 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:01:27.926400 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:01:27.929178 (+ 2778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:01:27.933073 (+ 3895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:01:27.933087 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:01:27.933111 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:01:27.933864 (+ 753us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:01:27.934396 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:01:27.934404 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:01:27.934409 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:01:27.934685 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:01:27.935267 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:01:27.935278 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:01:27.935804 (+ 526us) client_negotiation.cc:769] Sending connection context
0928 15:01:27.936188 (+ 384us) client_negotiation.cc:240] Negotiation successful
0928 15:01:27.936821 (+ 633us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":319,"threads_started":1}
I20260928 15:01:27.940129 19188 heartbeater.cc:347] Connected to a master server at 127.31.40.62:42437
I20260928 15:01:27.940522 19188 heartbeater.cc:464] Registering TS with master...
I20260928 15:01:27.941399 19188 heartbeater.cc:511] Master 127.31.40.62:42437 requested a full tablet report, sending...
I20260928 15:01:27.943120 18865 ts_manager.cc:194] Registered new tserver with Master: 32855210707a418495e2e2e139bf422e (127.31.40.2:34419)
I20260928 15:01:27.944499 18865 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.2@KRBTEST.COM'} at 127.31.40.2:44063
WARNING: no policy specified for kudu/127.31.40.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab.
Entry for principal kudu/127.31.40.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab.
I20260928 15:01:27.983618 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.3:0
--local_ip_for_outbound_sockets=127.31.40.3
--webserver_interface=127.31.40.3
--webserver_port=0
--tserver_master_addrs=127.31.40.62:42437
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab
--principal=kudu/127.31.40.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:43107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:01:28.334003 19194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:28.334374 19194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:28.334542 19194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:28.342511 19194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:28.342703 19194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:28.342891 19194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.3
I20260928 15:01:28.353523 19194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:43107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.3.keytab
--principal=kudu/127.31.40.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.40.3
--webserver_port=0
--tserver_master_addrs=127.31.40.62:42437
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.3
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:01:28.354943 19194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:28.356514 19194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:28.370729 19199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:28.370838 19200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:28.370841 19202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:28.371522 19194 server_base.cc:1034] running on GCE node
I20260928 15:01:28.372236 19194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:28.373322 19194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:28.374575 19194 hybrid_clock.cc:648] HybridClock initialized: now 1790607688374520 us; error 56 us; skew 500 ppm
I20260928 15:01:28.374948 19194 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:01:28 dist-test-slave-j968 krb5kdc[18820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607688, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:01:28.378420 19194 init.cc:377] Logged in from keytab as kudu/127.31.40.3@KRBTEST.COM (short username kudu)
I20260928 15:01:28.380299 19194 webserver.cc:460] Webserver started at http://127.31.40.3:34293/ using document root <none> and password file <none>
I20260928 15:01:28.381093 19194 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:28.381237 19194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:28.381569 19194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:28.384681 19194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/ts-2/data/instance:
uuid: "9cd560856bbd4fb5bba5ff5e54caefc2"
format_stamp: "Formatted at 2026-09-28 15:01:28 on dist-test-slave-j968"
server_key: "da3981e2e7284b2230149fbadc2fd82d"
server_key_iv: "88bca216302796107240191742634411"
server_key_version: "encryptionkey@0"
I20260928 15:01:28.385509 19194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/ts-2/wal/instance:
uuid: "9cd560856bbd4fb5bba5ff5e54caefc2"
format_stamp: "Formatted at 2026-09-28 15:01:28 on dist-test-slave-j968"
server_key: "da3981e2e7284b2230149fbadc2fd82d"
server_key_iv: "88bca216302796107240191742634411"
server_key_version: "encryptionkey@0"
I20260928 15:01:28.391127 19194 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 15:01:28.395547 19209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:28.397185 19194 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:01:28.397450 19194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/ts-2/wal
uuid: "9cd560856bbd4fb5bba5ff5e54caefc2"
format_stamp: "Formatted at 2026-09-28 15:01:28 on dist-test-slave-j968"
server_key: "da3981e2e7284b2230149fbadc2fd82d"
server_key_iv: "88bca216302796107240191742634411"
server_key_version: "encryptionkey@0"
I20260928 15:01:28.397718 19194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:28.398171 19054 heartbeater.cc:503] Master 127.31.40.62:42437 was elected leader, sending a full tablet report...
I20260928 15:01:28.423555 19194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:28.428964 19194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:28.429247 19194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:28.430572 19194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:01:28.432145 19194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:01:28.432230 19194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:28.432320 19194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:01:28.432364 19194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:28.468994 19194 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.3:36629
I20260928 15:01:28.469050 19321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.3:36629 every 8 connection(s)
I20260928 15:01:28.470418 19194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
I20260928 15:01:28.470974 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 19194
I20260928 15:01:28.471168 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/ts-2/wal/instance
I20260928 15:01:28.471787 31904 external_mini_cluster.cc:1428] Setting key f013abc8cd0261081a3eb590f605f207
Sep 28 15:01:28 dist-test-slave-j968 krb5kdc[18820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607688, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.3@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 15:01:28.492638 19324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:28.473511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:45139 (local address 127.31.40.62:42437)
0928 15:01:28.474144 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:28.474149 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:28.475265 (+ 1116us) server_negotiation.cc:407] Connection header received
0928 15:01:28.477179 (+ 1914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:28.477184 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:28.477284 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:28.477407 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:28.479101 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:28.480086 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:28.481741 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:28.482160 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:28.482304 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:28.486157 (+ 3853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:28.486162 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:28.486166 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:28.486203 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:28.489233 (+ 3030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:28.490043 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:28.490050 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:28.490054 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:28.490186 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:28.490653 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:28.490661 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:28.490664 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:28.490967 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:28.491147 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:28.491947 (+ 800us) server_negotiation.cc:299] Negotiation successful
0928 15:01:28.492193 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":382,"threads_started":1}
I20260928 15:01:28.493670 19325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:28.474224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:42437 (local address 127.31.40.3:45139)
0928 15:01:28.475113 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:28.475153 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:28.476919 (+ 1766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:28.477651 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:28.477660 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:28.478216 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:01:28.478840 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:28.478859 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:28.480252 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:28.480256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:28.481580 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:28.481593 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:28.482392 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:28.482399 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:28.482638 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:28.483341 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:01:28.483368 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:01:28.485901 (+ 2533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:01:28.489398 (+ 3497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:01:28.489417 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:01:28.489441 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:01:28.489908 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:01:28.490322 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:01:28.490327 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:01:28.490331 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:01:28.490533 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:01:28.491106 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:01:28.491115 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:01:28.491498 (+ 383us) client_negotiation.cc:769] Sending connection context
0928 15:01:28.491812 (+ 314us) client_negotiation.cc:240] Negotiation successful
0928 15:01:28.492446 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":314,"threads_started":1}
I20260928 15:01:28.495397 19322 heartbeater.cc:347] Connected to a master server at 127.31.40.62:42437
I20260928 15:01:28.495738 19322 heartbeater.cc:464] Registering TS with master...
I20260928 15:01:28.496524 19322 heartbeater.cc:511] Master 127.31.40.62:42437 requested a full tablet report, sending...
I20260928 15:01:28.498251 18865 ts_manager.cc:194] Registered new tserver with Master: 9cd560856bbd4fb5bba5ff5e54caefc2 (127.31.40.3:36629)
I20260928 15:01:28.499349 18865 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.40.3@KRBTEST.COM'} at 127.31.40.3:45139
I20260928 15:01:28.500524 31904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:01:28.519021 19324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:28.505601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60328 (local address 127.31.40.62:42437)
0928 15:01:28.505854 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:28.505864 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:28.506287 (+ 423us) server_negotiation.cc:407] Connection header received
0928 15:01:28.506486 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:28.506490 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:28.506557 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:28.506666 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:28.508124 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:28.508865 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:28.510215 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:28.510662 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:28.510803 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:28.513384 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:28.513388 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:28.513392 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:28.513424 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:28.516195 (+ 2771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:28.516973 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:28.516979 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:28.516984 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:28.517112 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:28.517503 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:28.517511 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:28.517514 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:28.517853 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:28.518120 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:28.518465 (+ 345us) server_negotiation.cc:299] Negotiation successful
0928 15:01:28.518691 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260928 15:01:28.524127 18864 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60328:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:01:28.531424 18864 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:01:28.556167 19337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:28.544630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44568 (local address 127.31.40.2:34419)
0928 15:01:28.547046 (+ 2416us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:28.547053 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:28.547076 (+ 23us) server_negotiation.cc:407] Connection header received
0928 15:01:28.547151 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:28.547157 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:28.547447 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:28.548405 (+ 958us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:28.548668 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:28.549863 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:28.553445 (+ 3582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:28.554660 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:28.554819 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:28.555045 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:28.555545 (+ 500us) server_negotiation.cc:299] Negotiation successful
0928 15:01:28.555744 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2259,"thread_start_us":2150,"threads_started":1}
I20260928 15:01:28.556996 19336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:28.544431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.2:34419 (local address 127.0.0.1:44568)
0928 15:01:28.545836 (+ 1405us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:28.545887 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:28.546096 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:28.547676 (+ 1580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:28.547684 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:28.547717 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:28.548211 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:28.548224 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:28.549998 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:28.550007 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:28.553253 (+ 3246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:28.553268 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:28.554887 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:28.554895 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:28.555180 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:28.555314 (+ 134us) client_negotiation.cc:769] Sending connection context
0928 15:01:28.555946 (+ 632us) client_negotiation.cc:240] Negotiation successful
0928 15:01:28.556165 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1068,"mutex_wait_us":57,"thread_start_us":2018,"threads_started":1}
I20260928 15:01:28.558920 19340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:28.548604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:43049 (local address 127.0.0.1:34156)
0928 15:01:28.550889 (+ 2285us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:28.550909 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:28.551045 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:28.552077 (+ 1032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:28.552085 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:28.552112 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:28.552457 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:28.552465 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:28.554067 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:28.554074 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:28.556296 (+ 2222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:28.556310 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:28.558058 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:28.558065 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:28.558252 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:28.558286 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 15:01:28.558383 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 15:01:28.558518 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2152,"mutex_wait_us":23,"thread_start_us":1985,"threads_started":1}
I20260928 15:01:28.559012 19339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:28.548309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34156 (local address 127.31.40.1:43049)
0928 15:01:28.548953 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:28.548962 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:28.551200 (+ 2238us) server_negotiation.cc:407] Connection header received
0928 15:01:28.551323 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:28.551330 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:28.551597 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:28.551814 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:28.552639 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:28.553911 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:28.556487 (+ 2576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:28.557820 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:28.557966 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:28.558203 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:28.558478 (+ 275us) server_negotiation.cc:299] Negotiation successful
0928 15:01:28.558657 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"spinlock_wait_cycles":87424,"thread_start_us":426,"threads_started":1}
I20260928 15:01:28.559746 19335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:28.544638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.3:36629 (local address 127.0.0.1:40388)
0928 15:01:28.546693 (+ 2055us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:28.546716 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:28.546834 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:28.551164 (+ 4330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:28.551170 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:28.551194 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:28.551930 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:28.551945 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:28.553793 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:28.553800 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:28.556805 (+ 3005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:28.556825 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:28.558935 (+ 2110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:28.558943 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:28.559134 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:28.559170 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 15:01:28.559262 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 15:01:28.559403 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1934,"mutex_wait_us":24,"thread_start_us":1021,"threads_started":1}
I20260928 15:01:28.560364 19338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:28.545325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40388 (local address 127.31.40.3:36629)
0928 15:01:28.550503 (+ 5178us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:28.550515 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:28.550551 (+ 36us) server_negotiation.cc:407] Connection header received
0928 15:01:28.550649 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:28.550657 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:28.550941 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:28.551297 (+ 356us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:28.552195 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:28.553667 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:28.556983 (+ 3316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:28.558440 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:28.558562 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:28.558783 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:28.559635 (+ 852us) server_negotiation.cc:299] Negotiation successful
0928 15:01:28.560014 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4992,"thread_start_us":4827,"threads_started":1}
I20260928 15:01:28.561586 19123 tablet_service.cc:1467] Processing CreateTablet for tablet 2f80e1fc71bf4e2ebb40d3f98aaf6822 (DEFAULT_TABLE table=test-table [id=77ea0f0afed441f99a362c17be8c164a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:01:28.562594 18989 tablet_service.cc:1467] Processing CreateTablet for tablet 2f80e1fc71bf4e2ebb40d3f98aaf6822 (DEFAULT_TABLE table=test-table [id=77ea0f0afed441f99a362c17be8c164a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:01:28.563452 19123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f80e1fc71bf4e2ebb40d3f98aaf6822. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:28.563804 18989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f80e1fc71bf4e2ebb40d3f98aaf6822. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:28.564284 19257 tablet_service.cc:1467] Processing CreateTablet for tablet 2f80e1fc71bf4e2ebb40d3f98aaf6822 (DEFAULT_TABLE table=test-table [id=77ea0f0afed441f99a362c17be8c164a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:01:28.565645 19257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f80e1fc71bf4e2ebb40d3f98aaf6822. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:28.576411 19341 tablet_bootstrap.cc:492] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P a43673a7ceb443e8895168abcf255ea9: Bootstrap starting.
I20260928 15:01:28.578827 19342 tablet_bootstrap.cc:492] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P 32855210707a418495e2e2e139bf422e: Bootstrap starting.
I20260928 15:01:28.580615 19341 tablet_bootstrap.cc:654] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P a43673a7ceb443e8895168abcf255ea9: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:28.581400 19343 tablet_bootstrap.cc:492] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P 9cd560856bbd4fb5bba5ff5e54caefc2: Bootstrap starting.
I20260928 15:01:28.582579 19341 log.cc:824] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P a43673a7ceb443e8895168abcf255ea9: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:28.582906 19342 tablet_bootstrap.cc:654] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P 32855210707a418495e2e2e139bf422e: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:28.584285 19342 log.cc:824] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P 32855210707a418495e2e2e139bf422e: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:28.585266 19343 tablet_bootstrap.cc:654] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P 9cd560856bbd4fb5bba5ff5e54caefc2: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:28.586584 19343 log.cc:824] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P 9cd560856bbd4fb5bba5ff5e54caefc2: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:28.587390 19341 tablet_bootstrap.cc:492] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P a43673a7ceb443e8895168abcf255ea9: No bootstrap required, opened a new log
I20260928 15:01:28.587702 19341 ts_tablet_manager.cc:1397] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P a43673a7ceb443e8895168abcf255ea9: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.003s
I20260928 15:01:28.587797 19342 tablet_bootstrap.cc:492] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P 32855210707a418495e2e2e139bf422e: No bootstrap required, opened a new log
I20260928 15:01:28.588111 19342 ts_tablet_manager.cc:1397] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P 32855210707a418495e2e2e139bf422e: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260928 15:01:28.589787 19343 tablet_bootstrap.cc:492] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P 9cd560856bbd4fb5bba5ff5e54caefc2: No bootstrap required, opened a new log
I20260928 15:01:28.590045 19343 ts_tablet_manager.cc:1397] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P 9cd560856bbd4fb5bba5ff5e54caefc2: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 15:01:28.595451 19341 raft_consensus.cc:348] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P a43673a7ceb443e8895168abcf255ea9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32855210707a418495e2e2e139bf422e" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34419 } } peers { permanent_uuid: "9cd560856bbd4fb5bba5ff5e54caefc2" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 36629 } } peers { permanent_uuid: "a43673a7ceb443e8895168abcf255ea9" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43049 } }
I20260928 15:01:28.595983 19341 raft_consensus.cc:374] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P a43673a7ceb443e8895168abcf255ea9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:28.596089 19341 raft_consensus.cc:729] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P a43673a7ceb443e8895168abcf255ea9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a43673a7ceb443e8895168abcf255ea9, State: Initialized, Role: FOLLOWER
I20260928 15:01:28.596724 19341 consensus_queue.cc:260] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P a43673a7ceb443e8895168abcf255ea9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32855210707a418495e2e2e139bf422e" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34419 } } peers { permanent_uuid: "9cd560856bbd4fb5bba5ff5e54caefc2" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 36629 } } peers { permanent_uuid: "a43673a7ceb443e8895168abcf255ea9" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43049 } }
I20260928 15:01:28.598562 19341 ts_tablet_manager.cc:1428] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P a43673a7ceb443e8895168abcf255ea9: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260928 15:01:28.599246 19342 raft_consensus.cc:348] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P 32855210707a418495e2e2e139bf422e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32855210707a418495e2e2e139bf422e" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34419 } } peers { permanent_uuid: "9cd560856bbd4fb5bba5ff5e54caefc2" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 36629 } } peers { permanent_uuid: "a43673a7ceb443e8895168abcf255ea9" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43049 } }
I20260928 15:01:28.599725 19342 raft_consensus.cc:374] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P 32855210707a418495e2e2e139bf422e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:28.599825 19342 raft_consensus.cc:729] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P 32855210707a418495e2e2e139bf422e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32855210707a418495e2e2e139bf422e, State: Initialized, Role: FOLLOWER
I20260928 15:01:28.600502 19342 consensus_queue.cc:260] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P 32855210707a418495e2e2e139bf422e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32855210707a418495e2e2e139bf422e" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34419 } } peers { permanent_uuid: "9cd560856bbd4fb5bba5ff5e54caefc2" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 36629 } } peers { permanent_uuid: "a43673a7ceb443e8895168abcf255ea9" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43049 } }
I20260928 15:01:28.601272 19343 raft_consensus.cc:348] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P 9cd560856bbd4fb5bba5ff5e54caefc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32855210707a418495e2e2e139bf422e" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34419 } } peers { permanent_uuid: "9cd560856bbd4fb5bba5ff5e54caefc2" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 36629 } } peers { permanent_uuid: "a43673a7ceb443e8895168abcf255ea9" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43049 } }
I20260928 15:01:28.601688 19188 heartbeater.cc:503] Master 127.31.40.62:42437 was elected leader, sending a full tablet report...
I20260928 15:01:28.601699 19343 raft_consensus.cc:374] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P 9cd560856bbd4fb5bba5ff5e54caefc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:28.601821 19343 raft_consensus.cc:729] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P 9cd560856bbd4fb5bba5ff5e54caefc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cd560856bbd4fb5bba5ff5e54caefc2, State: Initialized, Role: FOLLOWER
I20260928 15:01:28.602536 19342 ts_tablet_manager.cc:1428] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P 32855210707a418495e2e2e139bf422e: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260928 15:01:28.602435 19343 consensus_queue.cc:260] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P 9cd560856bbd4fb5bba5ff5e54caefc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32855210707a418495e2e2e139bf422e" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34419 } } peers { permanent_uuid: "9cd560856bbd4fb5bba5ff5e54caefc2" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 36629 } } peers { permanent_uuid: "a43673a7ceb443e8895168abcf255ea9" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43049 } }
I20260928 15:01:28.603595 19322 heartbeater.cc:503] Master 127.31.40.62:42437 was elected leader, sending a full tablet report...
I20260928 15:01:28.604305 19343 ts_tablet_manager.cc:1428] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P 9cd560856bbd4fb5bba5ff5e54caefc2: Time spent starting tablet: real 0.014s user 0.010s sys 0.005s
W20260928 15:01:28.610776 19055 tablet.cc:2369] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P a43673a7ceb443e8895168abcf255ea9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:01:28.663558 19189 tablet.cc:2369] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P 32855210707a418495e2e2e139bf422e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:01:28.699633 19347 raft_consensus.cc:482] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P a43673a7ceb443e8895168abcf255ea9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:01:28.700053 19347 raft_consensus.cc:504] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P a43673a7ceb443e8895168abcf255ea9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32855210707a418495e2e2e139bf422e" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34419 } } peers { permanent_uuid: "9cd560856bbd4fb5bba5ff5e54caefc2" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 36629 } } peers { permanent_uuid: "a43673a7ceb443e8895168abcf255ea9" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43049 } }
I20260928 15:01:28.702129 19347 leader_election.cc:288] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P a43673a7ceb443e8895168abcf255ea9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 32855210707a418495e2e2e139bf422e (127.31.40.2:34419), 9cd560856bbd4fb5bba5ff5e54caefc2 (127.31.40.3:36629)
I20260928 15:01:28.711822 19338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:28.702750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:45175 (local address 127.31.40.3:36629)
0928 15:01:28.702999 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:28.703011 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:28.703711 (+ 700us) server_negotiation.cc:407] Connection header received
0928 15:01:28.703956 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:28.703962 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:28.704042 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:28.704141 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:28.705064 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:28.706187 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:28.709246 (+ 3059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:28.710544 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:28.710696 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:28.710807 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:28.711430 (+ 623us) server_negotiation.cc:299] Negotiation successful
0928 15:01:28.711570 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 15:01:28.711822 19337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:28.702635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:47617 (local address 127.31.40.2:34419)
0928 15:01:28.702869 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:28.702875 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:28.703721 (+ 846us) server_negotiation.cc:407] Connection header received
0928 15:01:28.703956 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:28.703962 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:28.704042 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:28.704141 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:28.705064 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:28.706187 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:28.709239 (+ 3052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:28.710544 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:28.710696 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:28.710806 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:28.711403 (+ 597us) server_negotiation.cc:299] Negotiation successful
0928 15:01:28.711547 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 15:01:28.712107 19351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:28.702406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.2:34419 (local address 127.31.40.1:47617)
0928 15:01:28.703562 (+ 1156us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:28.703584 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:28.703768 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:28.704196 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:28.704201 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:28.704261 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:28.704902 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:28.704911 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:28.706369 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:28.706374 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:28.708993 (+ 2619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:28.709014 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:28.710901 (+ 1887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:28.710910 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:28.711186 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:28.711262 (+ 76us) client_negotiation.cc:769] Sending connection context
0928 15:01:28.711357 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 15:01:28.711551 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":927,"mutex_wait_us":107,"spinlock_wait_cycles":46976,"thread_start_us":894,"threads_started":1}
I20260928 15:01:28.712817 19350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:28.702558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.3:36629 (local address 127.31.40.1:45175)
0928 15:01:28.703600 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:28.703619 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:28.703768 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:28.704211 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:28.704215 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:28.704243 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:28.704844 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:28.704861 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:28.706368 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:28.706374 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:28.708993 (+ 2619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:28.709014 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:28.710923 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:28.710929 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:28.711186 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:28.711241 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 15:01:28.711357 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 15:01:28.711551 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":935,"mutex_wait_us":159,"spinlock_wait_cycles":34176,"thread_start_us":546,"threads_started":1}
I20260928 15:01:28.713286 19143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f80e1fc71bf4e2ebb40d3f98aaf6822" candidate_uuid: "a43673a7ceb443e8895168abcf255ea9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32855210707a418495e2e2e139bf422e" is_pre_election: true
I20260928 15:01:28.713973 19143 raft_consensus.cc:2454] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P 32855210707a418495e2e2e139bf422e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a43673a7ceb443e8895168abcf255ea9 in term 0.
I20260928 15:01:28.713900 19277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f80e1fc71bf4e2ebb40d3f98aaf6822" candidate_uuid: "a43673a7ceb443e8895168abcf255ea9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cd560856bbd4fb5bba5ff5e54caefc2" is_pre_election: true
I20260928 15:01:28.714498 19277 raft_consensus.cc:2454] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P 9cd560856bbd4fb5bba5ff5e54caefc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a43673a7ceb443e8895168abcf255ea9 in term 0.
I20260928 15:01:28.714754 18945 leader_election.cc:302] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P a43673a7ceb443e8895168abcf255ea9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32855210707a418495e2e2e139bf422e, a43673a7ceb443e8895168abcf255ea9; no voters:
I20260928 15:01:28.715212 19347 raft_consensus.cc:2781] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P a43673a7ceb443e8895168abcf255ea9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:01:28.715373 19347 raft_consensus.cc:482] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P a43673a7ceb443e8895168abcf255ea9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:01:28.715459 19347 raft_consensus.cc:3037] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P a43673a7ceb443e8895168abcf255ea9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:28.717694 19347 raft_consensus.cc:504] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P a43673a7ceb443e8895168abcf255ea9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32855210707a418495e2e2e139bf422e" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34419 } } peers { permanent_uuid: "9cd560856bbd4fb5bba5ff5e54caefc2" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 36629 } } peers { permanent_uuid: "a43673a7ceb443e8895168abcf255ea9" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43049 } }
I20260928 15:01:28.718339 19347 leader_election.cc:288] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P a43673a7ceb443e8895168abcf255ea9 [CANDIDATE]: Term 1 election: Requested vote from peers 32855210707a418495e2e2e139bf422e (127.31.40.2:34419), 9cd560856bbd4fb5bba5ff5e54caefc2 (127.31.40.3:36629)
I20260928 15:01:28.718915 19277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f80e1fc71bf4e2ebb40d3f98aaf6822" candidate_uuid: "a43673a7ceb443e8895168abcf255ea9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cd560856bbd4fb5bba5ff5e54caefc2"
I20260928 15:01:28.719007 19143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f80e1fc71bf4e2ebb40d3f98aaf6822" candidate_uuid: "a43673a7ceb443e8895168abcf255ea9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32855210707a418495e2e2e139bf422e"
I20260928 15:01:28.719131 19277 raft_consensus.cc:3037] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P 9cd560856bbd4fb5bba5ff5e54caefc2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:28.719286 19143 raft_consensus.cc:3037] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P 32855210707a418495e2e2e139bf422e [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:28.722179 19277 raft_consensus.cc:2454] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P 9cd560856bbd4fb5bba5ff5e54caefc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a43673a7ceb443e8895168abcf255ea9 in term 1.
I20260928 15:01:28.722180 19143 raft_consensus.cc:2454] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P 32855210707a418495e2e2e139bf422e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a43673a7ceb443e8895168abcf255ea9 in term 1.
W20260928 15:01:28.722463 19323 tablet.cc:2369] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P 9cd560856bbd4fb5bba5ff5e54caefc2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:01:28.722748 18945 leader_election.cc:302] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P a43673a7ceb443e8895168abcf255ea9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32855210707a418495e2e2e139bf422e, a43673a7ceb443e8895168abcf255ea9; no voters:
I20260928 15:01:28.723134 19347 raft_consensus.cc:2781] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P a43673a7ceb443e8895168abcf255ea9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:01:28.723632 19347 raft_consensus.cc:686] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P a43673a7ceb443e8895168abcf255ea9 [term 1 LEADER]: Becoming Leader. State: Replica: a43673a7ceb443e8895168abcf255ea9, State: Running, Role: LEADER
I20260928 15:01:28.724309 19347 consensus_queue.cc:237] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P a43673a7ceb443e8895168abcf255ea9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32855210707a418495e2e2e139bf422e" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34419 } } peers { permanent_uuid: "9cd560856bbd4fb5bba5ff5e54caefc2" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 36629 } } peers { permanent_uuid: "a43673a7ceb443e8895168abcf255ea9" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43049 } }
I20260928 15:01:28.729547 18864 catalog_manager.cc:5495] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P a43673a7ceb443e8895168abcf255ea9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a43673a7ceb443e8895168abcf255ea9 (127.31.40.1). New cstate: current_term: 1 leader_uuid: "a43673a7ceb443e8895168abcf255ea9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32855210707a418495e2e2e139bf422e" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34419 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cd560856bbd4fb5bba5ff5e54caefc2" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 36629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a43673a7ceb443e8895168abcf255ea9" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 43049 } health_report { overall_health: HEALTHY } } }
I20260928 15:01:28.748723 19339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:28.743637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34170 (local address 127.31.40.1:43049)
0928 15:01:28.743820 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:28.743824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:28.743841 (+ 17us) server_negotiation.cc:407] Connection header received
0928 15:01:28.743984 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:28.743988 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:28.744070 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:28.744197 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:01:28.744569 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:28.745282 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:28.746630 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:28.747030 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:28.747122 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:28.747595 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:01:28.748195 (+ 600us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:01:28.748309 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:28.748450 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 15:01:28.748518 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 15:01:28.757563 19143 raft_consensus.cc:1261] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P 32855210707a418495e2e2e139bf422e [term 1 FOLLOWER]: Refusing update from remote peer a43673a7ceb443e8895168abcf255ea9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:01:28.757563 19277 raft_consensus.cc:1261] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P 9cd560856bbd4fb5bba5ff5e54caefc2 [term 1 FOLLOWER]: Refusing update from remote peer a43673a7ceb443e8895168abcf255ea9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:01:28.758641 19347 consensus_queue.cc:1035] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P a43673a7ceb443e8895168abcf255ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cd560856bbd4fb5bba5ff5e54caefc2" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 36629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:01:28.758951 19352 consensus_queue.cc:1035] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P a43673a7ceb443e8895168abcf255ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32855210707a418495e2e2e139bf422e" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 34419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:01:28.774562 19356 mvcc.cc:205] Tried to move back new op lower bound from 7334329093140475904 to 7334329093018619904. Current Snapshot: MvccSnapshot[applied={T|T < 7334329093140475904}]
I20260928 15:01:28.787468 18865 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:01:28.787753 18865 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:01:28.791622 18865 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 28647ebd484646b5ac7c03f702e24a18: Sending DeleteTablet for 3 replicas of tablet 2f80e1fc71bf4e2ebb40d3f98aaf6822
I20260928 15:01:28.793107 19257 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f80e1fc71bf4e2ebb40d3f98aaf6822 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:01:28 UTC) from {username='kudu', principal='kudu/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:40388
I20260928 15:01:28.793485 18989 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f80e1fc71bf4e2ebb40d3f98aaf6822 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:01:28 UTC) from {username='kudu', principal='kudu/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:34156
I20260928 15:01:28.793918 19123 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f80e1fc71bf4e2ebb40d3f98aaf6822 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:01:28 UTC) from {username='kudu', principal='kudu/127.31.40.62@KRBTEST.COM'} at 127.0.0.1:44568
I20260928 15:01:28.794215 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 18926
I20260928 15:01:28.810531 19367 tablet_replica.cc:330] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P 32855210707a418495e2e2e139bf422e: stopping tablet replica
I20260928 15:01:28.811204 19367 raft_consensus.cc:2229] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P 32855210707a418495e2e2e139bf422e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:01:28.811970 19367 raft_consensus.cc:2258] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P 32855210707a418495e2e2e139bf422e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:01:28.815584 19365 tablet_replica.cc:330] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P 9cd560856bbd4fb5bba5ff5e54caefc2: stopping tablet replica
I20260928 15:01:28.816232 19365 raft_consensus.cc:2229] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P 9cd560856bbd4fb5bba5ff5e54caefc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:01:28.816802 19365 raft_consensus.cc:2258] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P 9cd560856bbd4fb5bba5ff5e54caefc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:01:28.818568 19367 ts_tablet_manager.cc:1905] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P 32855210707a418495e2e2e139bf422e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:01:28.819164 19365 ts_tablet_manager.cc:1905] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P 9cd560856bbd4fb5bba5ff5e54caefc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:01:28.827111 19365 ts_tablet_manager.cc:1918] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P 9cd560856bbd4fb5bba5ff5e54caefc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:01:28.827339 19365 log.cc:1196] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P 9cd560856bbd4fb5bba5ff5e54caefc2: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/ts-2/wal/wals/2f80e1fc71bf4e2ebb40d3f98aaf6822
Sep 28 15:01:28 dist-test-slave-j968 krb5kdc[18820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.1@KRBTEST.COM
I20260928 15:01:28.827965 19365 ts_tablet_manager.cc:1939] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P 9cd560856bbd4fb5bba5ff5e54caefc2: Deleting consensus metadata
I20260928 15:01:28.828169 19366 tablet_replica.cc:330] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P a43673a7ceb443e8895168abcf255ea9: stopping tablet replica
I20260928 15:01:28.828696 19366 raft_consensus.cc:2229] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P a43673a7ceb443e8895168abcf255ea9 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:01:28.829414 19366 raft_consensus.cc:2258] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P a43673a7ceb443e8895168abcf255ea9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:01:28.830168 18850 catalog_manager.cc:4841] TS 9cd560856bbd4fb5bba5ff5e54caefc2 (127.31.40.3:36629): tablet 2f80e1fc71bf4e2ebb40d3f98aaf6822 (table test-table [id=77ea0f0afed441f99a362c17be8c164a]) successfully deleted
I20260928 15:01:28.832435 19366 ts_tablet_manager.cc:1905] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P a43673a7ceb443e8895168abcf255ea9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:01:28.833917 19367 ts_tablet_manager.cc:1918] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P 32855210707a418495e2e2e139bf422e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:01:28.834110 19367 log.cc:1196] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P 32855210707a418495e2e2e139bf422e: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/ts-1/wal/wals/2f80e1fc71bf4e2ebb40d3f98aaf6822
I20260928 15:01:28.834681 19367 ts_tablet_manager.cc:1939] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P 32855210707a418495e2e2e139bf422e: Deleting consensus metadata
I20260928 15:01:28.836709 18852 catalog_manager.cc:4841] TS 32855210707a418495e2e2e139bf422e (127.31.40.2:34419): tablet 2f80e1fc71bf4e2ebb40d3f98aaf6822 (table test-table [id=77ea0f0afed441f99a362c17be8c164a]) successfully deleted
I20260928 15:01:28.839673 19339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:28.795893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34178 (local address 127.31.40.1:43049)
0928 15:01:28.796143 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:28.796152 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:28.806291 (+ 10139us) server_negotiation.cc:407] Connection header received
0928 15:01:28.807516 (+ 1225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:28.807526 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:28.807649 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:28.807735 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:28.812480 (+ 4745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:28.813988 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:28.821639 (+ 7651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:28.822475 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:28.822591 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:28.830798 (+ 8207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:28.830814 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:28.830834 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:28.830884 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:28.835387 (+ 4503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:28.836542 (+ 1155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:28.836567 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:28.836570 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:28.836791 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:28.837417 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:28.837424 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:28.837428 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:28.837930 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:28.838311 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:28.838962 (+ 651us) server_negotiation.cc:299] Negotiation successful
0928 15:01:28.839251 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 15:01:28.840466 19049 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:01:28.941067 19366 ts_tablet_manager.cc:1918] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P a43673a7ceb443e8895168abcf255ea9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:01:28.941366 19366 log.cc:1196] T 2f80e1fc71bf4e2ebb40d3f98aaf6822 P a43673a7ceb443e8895168abcf255ea9: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607365362981-31904-0/minicluster-data/ts-0/wal/wals/2f80e1fc71bf4e2ebb40d3f98aaf6822
I20260928 15:01:28.941737 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 18926
W20260928 15:01:28.961158 18852 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.40.1:43049 (error 108)
I20260928 15:01:28.961354 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 19060
I20260928 15:01:28.963416 19335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:28.962886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:43049 (local address 127.0.0.1:34184)
0928 15:01:28.963085 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:28.963221 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.1:43049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 15:01:28.963752 18852 catalog_manager.cc:4584] TS a43673a7ceb443e8895168abcf255ea9 (127.31.40.1:43049): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2f80e1fc71bf4e2ebb40d3f98aaf6822: Network error: Client connection negotiation failed: client connection to 127.31.40.1:43049: connect: Connection refused (error 111)
Sep 28 15:01:28 dist-test-slave-j968 krb5kdc[18820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.2@KRBTEST.COM
I20260928 15:01:28.980011 19337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:28.962219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44578 (local address 127.31.40.2:34419)
0928 15:01:28.962410 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:28.962415 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:28.962642 (+ 227us) server_negotiation.cc:407] Connection header received
0928 15:01:28.962924 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:28.962931 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:28.963033 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:28.963111 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:28.964897 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:28.965687 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:28.967156 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:28.967573 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:28.967677 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:28.973233 (+ 5556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:28.973252 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:28.973271 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:28.973325 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:28.976933 (+ 3608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:28.977856 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:28.977866 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:28.977868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:28.978026 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:28.978409 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:28.978413 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:28.978415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:28.978917 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:28.979160 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:28.979421 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 15:01:28.979595 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 15:01:28.980770 19183 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:01:29.017690 19335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:29.017078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:43049 (local address 127.0.0.1:34196)
0928 15:01:29.017340 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:29.017468 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.1:43049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260928 15:01:29.018952 19335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:29.018538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:43049 (local address 127.0.0.1:34200)
0928 15:01:29.018711 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:29.018799 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.1:43049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 15:01:29.073911 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 19060
I20260928 15:01:29.079638 19335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:29.079143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:43049 (local address 127.0.0.1:34216)
0928 15:01:29.079374 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:29.079482 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.1:43049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260928 15:01:29.080672 19335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:29.080321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:43049 (local address 127.0.0.1:34218)
0928 15:01:29.080472 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:29.080549 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.1:43049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 15:01:29.092739 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 19194
Sep 28 15:01:29 dist-test-slave-j968 krb5kdc[18820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.3@KRBTEST.COM
I20260928 15:01:29.109313 19338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:29.093622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40396 (local address 127.31.40.3:36629)
0928 15:01:29.093790 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:29.093794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:29.093814 (+ 20us) server_negotiation.cc:407] Connection header received
0928 15:01:29.093983 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:29.093988 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:29.094058 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:29.094152 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:29.095491 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:29.096295 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:29.097671 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:29.098104 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:29.098202 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:29.103083 (+ 4881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:29.103101 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:29.103119 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:29.103158 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:29.106261 (+ 3103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:29.107059 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:29.107067 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:29.107069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:29.107207 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:29.107608 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:29.107613 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:29.107615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:29.108156 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:29.108412 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:29.108788 (+ 376us) server_negotiation.cc:299] Negotiation successful
0928 15:01:29.108984 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 15:01:29.110093 19317 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:01:29.161094 19335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:29.160563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:43049 (local address 127.0.0.1:34232)
0928 15:01:29.160792 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:29.160913 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.1:43049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260928 15:01:29.162360 19335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:29.161884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:43049 (local address 127.0.0.1:34240)
0928 15:01:29.162128 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:29.162220 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.1:43049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260928 15:01:29.202135 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 19194
I20260928 15:01:29.220100 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 18833
I20260928 15:01:29.221338 18895 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:01:29.336634 19335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:29.335979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:43049 (local address 127.0.0.1:34256)
0928 15:01:29.336338 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:29.336451 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.1:43049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 15:01:29.336784 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 18833
2026-09-28T15:01:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4540 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T15:01:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:01:29Z Disabled control of system clock
I20260928 15:01:29.420104 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:45727
--webserver_interface=127.31.40.62
--webserver_port=0
--builtin_ntp_servers=127.31.40.20:38513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.40.62:45727
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 15:01:29.771996 19377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:29.772320 19377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:29.772403 19377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:29.779925 19377 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:01:29.780043 19377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:29.780081 19377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:29.780138 19377 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:01:29.780180 19377 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:01:29.789623 19377 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:38513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.40.62:45727
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:45727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:01:29.790958 19377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:29.792874 19377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:29.805215 19382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:29.805181 19385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:29.805136 19383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:29.806231 19377 server_base.cc:1034] running on GCE node
I20260928 15:01:29.807098 19377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:29.808588 19377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:29.809873 19377 hybrid_clock.cc:648] HybridClock initialized: now 1790607689809798 us; error 57 us; skew 500 ppm
I20260928 15:01:29.810259 19377 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:01:29.812670 19377 webserver.cc:460] Webserver started at http://127.31.40.62:38673/ using document root <none> and password file <none>
I20260928 15:01:29.813372 19377 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:29.813464 19377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:29.813750 19377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:29.816910 19377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/master-0/data/instance:
uuid: "32064cce24694da6be3580143a0e4b28"
format_stamp: "Formatted at 2026-09-28 15:01:29 on dist-test-slave-j968"
server_key: "aaee1b1e02d5582dbd7e27b30fffa5cc"
server_key_iv: "ba605d395e0e2e0b33e58e2cb652d5dd"
server_key_version: "encryptionkey@0"
I20260928 15:01:29.817728 19377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/master-0/wal/instance:
uuid: "32064cce24694da6be3580143a0e4b28"
format_stamp: "Formatted at 2026-09-28 15:01:29 on dist-test-slave-j968"
server_key: "aaee1b1e02d5582dbd7e27b30fffa5cc"
server_key_iv: "ba605d395e0e2e0b33e58e2cb652d5dd"
server_key_version: "encryptionkey@0"
I20260928 15:01:29.823163 19377 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 15:01:29.827028 19391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:29.828681 19377 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 15:01:29.828979 19377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/master-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/master-0/wal
uuid: "32064cce24694da6be3580143a0e4b28"
format_stamp: "Formatted at 2026-09-28 15:01:29 on dist-test-slave-j968"
server_key: "aaee1b1e02d5582dbd7e27b30fffa5cc"
server_key_iv: "ba605d395e0e2e0b33e58e2cb652d5dd"
server_key_version: "encryptionkey@0"
I20260928 15:01:29.829232 19377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:29.842767 19377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:29.843677 19377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:29.844062 19377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:29.863890 19377 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.62:45727
I20260928 15:01:29.864027 19442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.62:45727 every 8 connection(s)
I20260928 15:01:29.865433 19377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
I20260928 15:01:29.867452 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 19377
I20260928 15:01:29.867632 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/master-0/wal/instance
I20260928 15:01:29.868213 31904 external_mini_cluster.cc:1428] Setting key 80c4313428ff720797540d9925d58fe6
I20260928 15:01:29.870261 19443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:29.882140 19446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:29.870221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40580 (local address 127.31.40.62:45727)
0928 15:01:29.871311 (+ 1090us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:29.871322 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:29.871363 (+ 41us) server_negotiation.cc:407] Connection header received
0928 15:01:29.873028 (+ 1665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:29.873051 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:29.873536 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:29.873958 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:29.874570 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:29.876057 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:29.877444 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:29.878150 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:29.878348 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:29.879602 (+ 1254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:29.879611 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:29.879627 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:01:29.880335 (+ 708us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:29.880413 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:01:29.880437 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:29.880580 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:29.880820 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 15:01:29.881141 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":357,"threads_started":1}
I20260928 15:01:29.882193 19443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32064cce24694da6be3580143a0e4b28: Bootstrap starting.
I20260928 15:01:29.885957 19443 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32064cce24694da6be3580143a0e4b28: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:29.887142 19443 log.cc:824] T 00000000000000000000000000000000 P 32064cce24694da6be3580143a0e4b28: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:29.890137 19443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32064cce24694da6be3580143a0e4b28: No bootstrap required, opened a new log
I20260928 15:01:29.897663 19443 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32064cce24694da6be3580143a0e4b28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32064cce24694da6be3580143a0e4b28" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 45727 } }
I20260928 15:01:29.898031 19443 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32064cce24694da6be3580143a0e4b28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:29.898105 19443 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32064cce24694da6be3580143a0e4b28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32064cce24694da6be3580143a0e4b28, State: Initialized, Role: FOLLOWER
I20260928 15:01:29.898680 19443 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32064cce24694da6be3580143a0e4b28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32064cce24694da6be3580143a0e4b28" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 45727 } }
I20260928 15:01:29.898893 19443 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32064cce24694da6be3580143a0e4b28 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:01:29.898993 19443 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32064cce24694da6be3580143a0e4b28 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:01:29.899123 19443 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32064cce24694da6be3580143a0e4b28 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:29.901813 19443 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32064cce24694da6be3580143a0e4b28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32064cce24694da6be3580143a0e4b28" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 45727 } }
I20260928 15:01:29.902280 19443 leader_election.cc:302] T 00000000000000000000000000000000 P 32064cce24694da6be3580143a0e4b28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32064cce24694da6be3580143a0e4b28; no voters:
I20260928 15:01:29.902823 19443 leader_election.cc:288] T 00000000000000000000000000000000 P 32064cce24694da6be3580143a0e4b28 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:01:29.903012 19448 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32064cce24694da6be3580143a0e4b28 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:01:29.903767 19448 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32064cce24694da6be3580143a0e4b28 [term 1 LEADER]: Becoming Leader. State: Replica: 32064cce24694da6be3580143a0e4b28, State: Running, Role: LEADER
I20260928 15:01:29.904361 19448 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32064cce24694da6be3580143a0e4b28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32064cce24694da6be3580143a0e4b28" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 45727 } }
I20260928 15:01:29.905031 19443 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32064cce24694da6be3580143a0e4b28 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:01:29.907266 19450 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32064cce24694da6be3580143a0e4b28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32064cce24694da6be3580143a0e4b28. Latest consensus state: current_term: 1 leader_uuid: "32064cce24694da6be3580143a0e4b28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32064cce24694da6be3580143a0e4b28" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 45727 } } }
I20260928 15:01:29.907763 19450 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32064cce24694da6be3580143a0e4b28 [sys.catalog]: This master's current role is: LEADER
I20260928 15:01:29.908560 19449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32064cce24694da6be3580143a0e4b28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32064cce24694da6be3580143a0e4b28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32064cce24694da6be3580143a0e4b28" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 45727 } } }
I20260928 15:01:29.908952 19449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32064cce24694da6be3580143a0e4b28 [sys.catalog]: This master's current role is: LEADER
I20260928 15:01:29.910112 19457 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:01:29.915133 19457 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:01:29.924314 19457 catalog_manager.cc:1347] Generated new cluster ID: 06b76c8c9f5e428ca79493ba006dab13
I20260928 15:01:29.924487 19457 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:01:29.938179 19457 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:01:29.940143 19457 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:01:29.950804 19457 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32064cce24694da6be3580143a0e4b28: Generated new TSK 0
I20260928 15:01:29.951967 19457 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:01:29.964284 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.1:0
--local_ip_for_outbound_sockets=127.31.40.1
--webserver_interface=127.31.40.1
--webserver_port=0
--tserver_master_addrs=127.31.40.62:45727
--builtin_ntp_servers=127.31.40.20:38513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 15:01:30.315255 19467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:30.315596 19467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:30.315769 19467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:30.323172 19467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:30.323342 19467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:30.323524 19467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.1
I20260928 15:01:30.333679 19467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:38513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.40.1
--webserver_port=0
--tserver_master_addrs=127.31.40.62:45727
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.1
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:01:30.335114 19467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:30.337064 19467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:30.350965 19472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:30.351191 19473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:30.351372 19475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:30.352161 19467 server_base.cc:1034] running on GCE node
I20260928 15:01:30.352810 19467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:30.353825 19467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:30.355082 19467 hybrid_clock.cc:648] HybridClock initialized: now 1790607690355025 us; error 48 us; skew 500 ppm
I20260928 15:01:30.355434 19467 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:01:30.357810 19467 webserver.cc:460] Webserver started at http://127.31.40.1:43783/ using document root <none> and password file <none>
I20260928 15:01:30.358525 19467 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:30.358659 19467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:30.358986 19467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:30.361976 19467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/ts-0/data/instance:
uuid: "8f6c2a3592a84c49ae47fde726dcfc26"
format_stamp: "Formatted at 2026-09-28 15:01:30 on dist-test-slave-j968"
server_key: "628949d8d86c35fc255e5288b7a452ea"
server_key_iv: "88ee19dbb6c4ce124cee8253da0c716b"
server_key_version: "encryptionkey@0"
I20260928 15:01:30.362792 19467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/ts-0/wal/instance:
uuid: "8f6c2a3592a84c49ae47fde726dcfc26"
format_stamp: "Formatted at 2026-09-28 15:01:30 on dist-test-slave-j968"
server_key: "628949d8d86c35fc255e5288b7a452ea"
server_key_iv: "88ee19dbb6c4ce124cee8253da0c716b"
server_key_version: "encryptionkey@0"
I20260928 15:01:30.368643 19467 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 15:01:30.372345 19481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:30.373749 19467 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 15:01:30.373955 19467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/ts-0/wal
uuid: "8f6c2a3592a84c49ae47fde726dcfc26"
format_stamp: "Formatted at 2026-09-28 15:01:30 on dist-test-slave-j968"
server_key: "628949d8d86c35fc255e5288b7a452ea"
server_key_iv: "88ee19dbb6c4ce124cee8253da0c716b"
server_key_version: "encryptionkey@0"
I20260928 15:01:30.374186 19467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:30.393927 19467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:30.394812 19467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:30.395133 19467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:30.396247 19467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:01:30.397781 19467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:01:30.397918 19467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:30.398046 19467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:01:30.398132 19467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:30.432062 19467 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.1:42215
I20260928 15:01:30.432114 19593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.1:42215 every 8 connection(s)
I20260928 15:01:30.433545 19467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/ts-0/data/info.pb
I20260928 15:01:30.441854 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 19467
I20260928 15:01:30.442055 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/ts-0/wal/instance
I20260928 15:01:30.442512 31904 external_mini_cluster.cc:1428] Setting key 48a363f2f2461fd60f7478a29d8e78c0
I20260928 15:01:30.445653 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.2:0
--local_ip_for_outbound_sockets=127.31.40.2
--webserver_interface=127.31.40.2
--webserver_port=0
--tserver_master_addrs=127.31.40.62:45727
--builtin_ntp_servers=127.31.40.20:38513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 15:01:30.447007 19596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:30.436238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:47477 (local address 127.31.40.62:45727)
0928 15:01:30.436760 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:30.436765 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:30.437824 (+ 1059us) server_negotiation.cc:407] Connection header received
0928 15:01:30.439653 (+ 1829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:30.439659 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:30.439758 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:30.439961 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:30.441049 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:30.441817 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:30.443783 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:30.444356 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:30.444479 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:30.445993 (+ 1514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:30.446002 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:30.446010 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:01:30.446035 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:30.446089 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:01:30.446107 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:30.446205 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:30.446431 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 15:01:30.446552 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":307,"threads_started":1}
I20260928 15:01:30.447959 19597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:30.436753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:45727 (local address 127.31.40.1:47477)
0928 15:01:30.437662 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:30.437701 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:30.439363 (+ 1662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:30.440171 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:30.440180 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:30.440276 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:01:30.440872 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:30.440891 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:30.441989 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:30.441996 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:30.443621 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:30.443637 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:30.444610 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:30.444620 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:30.444848 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:30.445622 (+ 774us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:01:30.445625 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:01:30.445667 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:01:30.445755 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:01:30.446210 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:01:30.446219 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:01:30.446225 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 15:01:30.446384 (+ 159us) client_negotiation.cc:240] Negotiation successful
0928 15:01:30.446711 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":281,"threads_started":1}
I20260928 15:01:30.449654 19594 heartbeater.cc:347] Connected to a master server at 127.31.40.62:45727
I20260928 15:01:30.450003 19594 heartbeater.cc:464] Registering TS with master...
I20260928 15:01:30.450850 19594 heartbeater.cc:511] Master 127.31.40.62:45727 requested a full tablet report, sending...
I20260928 15:01:30.453222 19408 ts_manager.cc:194] Registered new tserver with Master: 8f6c2a3592a84c49ae47fde726dcfc26 (127.31.40.1:42215)
I20260928 15:01:30.454808 19408 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.40.1:47477
W20260928 15:01:30.789609 19598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:30.789929 19598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:30.790078 19598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:30.797760 19598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:30.797892 19598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:30.798089 19598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.2
I20260928 15:01:30.808156 19598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:38513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.40.2
--webserver_port=0
--tserver_master_addrs=127.31.40.62:45727
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.2
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:01:30.809537 19598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:30.811480 19598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:30.826105 19604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:30.826025 19606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:30.826064 19603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:30.826895 19598 server_base.cc:1034] running on GCE node
I20260928 15:01:30.827548 19598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:30.828653 19598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:30.829907 19598 hybrid_clock.cc:648] HybridClock initialized: now 1790607690829846 us; error 46 us; skew 500 ppm
I20260928 15:01:30.830348 19598 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:01:30.833052 19598 webserver.cc:460] Webserver started at http://127.31.40.2:45199/ using document root <none> and password file <none>
I20260928 15:01:30.833860 19598 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:30.833978 19598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:30.834321 19598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:30.837453 19598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/ts-1/data/instance:
uuid: "bfc8697373634f65bc7ea8b04c8fea08"
format_stamp: "Formatted at 2026-09-28 15:01:30 on dist-test-slave-j968"
server_key: "859958cd9135f325f382eed8c82d5084"
server_key_iv: "e3bf1e8dee718794a818f4ed164b7b28"
server_key_version: "encryptionkey@0"
I20260928 15:01:30.838294 19598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/ts-1/wal/instance:
uuid: "bfc8697373634f65bc7ea8b04c8fea08"
format_stamp: "Formatted at 2026-09-28 15:01:30 on dist-test-slave-j968"
server_key: "859958cd9135f325f382eed8c82d5084"
server_key_iv: "e3bf1e8dee718794a818f4ed164b7b28"
server_key_version: "encryptionkey@0"
I20260928 15:01:30.844081 19598 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260928 15:01:30.847771 19612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:30.849522 19598 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:01:30.849743 19598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/ts-1/wal
uuid: "bfc8697373634f65bc7ea8b04c8fea08"
format_stamp: "Formatted at 2026-09-28 15:01:30 on dist-test-slave-j968"
server_key: "859958cd9135f325f382eed8c82d5084"
server_key_iv: "e3bf1e8dee718794a818f4ed164b7b28"
server_key_version: "encryptionkey@0"
I20260928 15:01:30.849989 19598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:30.873739 19598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:30.874614 19598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:30.874914 19598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:30.876025 19598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:01:30.877705 19598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:01:30.877835 19598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:30.877966 19598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:01:30.878052 19598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:30.913966 19598 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.2:37817
I20260928 15:01:30.914057 19724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.2:37817 every 8 connection(s)
I20260928 15:01:30.915380 19598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/ts-1/data/info.pb
I20260928 15:01:30.924482 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 19598
I20260928 15:01:30.924708 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/ts-1/wal/instance
I20260928 15:01:30.925227 31904 external_mini_cluster.cc:1428] Setting key afb372e7bb1fd90fd9a8c4f2e2077aae
I20260928 15:01:30.928159 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.40.3:0
--local_ip_for_outbound_sockets=127.31.40.3
--webserver_interface=127.31.40.3
--webserver_port=0
--tserver_master_addrs=127.31.40.62:45727
--builtin_ntp_servers=127.31.40.20:38513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 15:01:30.929946 19596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:30.918342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.2:49369 (local address 127.31.40.62:45727)
0928 15:01:30.918549 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:30.918554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:30.920199 (+ 1645us) server_negotiation.cc:407] Connection header received
0928 15:01:30.922241 (+ 2042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:30.922248 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:30.922358 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:30.922483 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:30.923758 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:30.924791 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:30.926731 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:30.927196 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:30.927354 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:30.929004 (+ 1650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:30.929010 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:30.929027 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:01:30.929048 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:30.929083 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:01:30.929097 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:30.929220 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:30.929482 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 15:01:30.929599 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 15:01:30.931155 19727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:30.918873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:45727 (local address 127.31.40.2:49369)
0928 15:01:30.920012 (+ 1139us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:30.920067 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:30.921916 (+ 1849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:30.922750 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:30.922759 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:30.922867 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:01:30.923543 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:30.923562 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:30.924981 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:30.924994 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:30.926542 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:30.926557 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:30.927466 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:30.927476 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:30.927736 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:30.928602 (+ 866us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:01:30.928604 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:01:30.928651 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:01:30.928746 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:01:30.929217 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:01:30.929232 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:01:30.929255 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 15:01:30.929470 (+ 215us) client_negotiation.cc:240] Negotiation successful
0928 15:01:30.929961 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":358,"threads_started":1}
I20260928 15:01:30.933059 19725 heartbeater.cc:347] Connected to a master server at 127.31.40.62:45727
I20260928 15:01:30.933430 19725 heartbeater.cc:464] Registering TS with master...
I20260928 15:01:30.934475 19725 heartbeater.cc:511] Master 127.31.40.62:45727 requested a full tablet report, sending...
I20260928 15:01:30.936244 19408 ts_manager.cc:194] Registered new tserver with Master: bfc8697373634f65bc7ea8b04c8fea08 (127.31.40.2:37817)
I20260928 15:01:30.937685 19408 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.40.2:49369
W20260928 15:01:31.272746 19728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:31.273057 19728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:31.273145 19728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:31.281013 19728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:31.281132 19728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:31.281241 19728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.40.3
I20260928 15:01:31.290825 19728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:38513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.40.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.40.3
--webserver_port=0
--tserver_master_addrs=127.31.40.62:45727
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.40.3
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:01:31.292128 19728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:31.293990 19728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:31.308203 19733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:31.308290 19734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:31.308246 19736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:31.309226 19728 server_base.cc:1034] running on GCE node
I20260928 15:01:31.309896 19728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:31.311009 19728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:31.312287 19728 hybrid_clock.cc:648] HybridClock initialized: now 1790607691312220 us; error 59 us; skew 500 ppm
I20260928 15:01:31.312691 19728 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:01:31.315616 19728 webserver.cc:460] Webserver started at http://127.31.40.3:35355/ using document root <none> and password file <none>
I20260928 15:01:31.316500 19728 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:31.316650 19728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:31.317003 19728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:31.320374 19728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/ts-2/data/instance:
uuid: "512260d042bd45d8885ecff5798f3809"
format_stamp: "Formatted at 2026-09-28 15:01:31 on dist-test-slave-j968"
server_key: "52aba9cc135212a1a23b72abe40fca61"
server_key_iv: "f0276cffb828780ec0d07c6f1ba6c4d4"
server_key_version: "encryptionkey@0"
I20260928 15:01:31.321310 19728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/ts-2/wal/instance:
uuid: "512260d042bd45d8885ecff5798f3809"
format_stamp: "Formatted at 2026-09-28 15:01:31 on dist-test-slave-j968"
server_key: "52aba9cc135212a1a23b72abe40fca61"
server_key_iv: "f0276cffb828780ec0d07c6f1ba6c4d4"
server_key_version: "encryptionkey@0"
I20260928 15:01:31.327396 19728 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260928 15:01:31.331399 19742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:31.333235 19728 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:01:31.333516 19728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/ts-2/wal
uuid: "512260d042bd45d8885ecff5798f3809"
format_stamp: "Formatted at 2026-09-28 15:01:31 on dist-test-slave-j968"
server_key: "52aba9cc135212a1a23b72abe40fca61"
server_key_iv: "f0276cffb828780ec0d07c6f1ba6c4d4"
server_key_version: "encryptionkey@0"
I20260928 15:01:31.333832 19728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:31.378151 19728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:31.379125 19728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:31.379418 19728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:31.380676 19728 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:01:31.382236 19728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:01:31.382360 19728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:31.382495 19728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:01:31.382591 19728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:31.418030 19728 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.3:38441
I20260928 15:01:31.418120 19854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.3:38441 every 8 connection(s)
I20260928 15:01:31.419415 19728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/ts-2/data/info.pb
I20260928 15:01:31.428357 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 19728
I20260928 15:01:31.428560 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/ts-2/wal/instance
I20260928 15:01:31.429148 31904 external_mini_cluster.cc:1428] Setting key 788183e63978388b88115881ce25e04b
I20260928 15:01:31.435050 19596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:31.422538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.3:50205 (local address 127.31.40.62:45727)
0928 15:01:31.422812 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:31.422817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:31.424416 (+ 1599us) server_negotiation.cc:407] Connection header received
0928 15:01:31.426265 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:31.426270 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:31.426399 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:31.426497 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:31.427832 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:31.428845 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:31.431418 (+ 2573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:31.432121 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:31.432268 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:31.434123 (+ 1855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:31.434129 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:31.434134 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:01:31.434160 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:31.434203 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:01:31.434219 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:31.434326 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:31.434626 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 15:01:31.434733 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 15:01:31.436265 19857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:31.423097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:45727 (local address 127.31.40.3:50205)
0928 15:01:31.424208 (+ 1111us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:31.424258 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:31.425944 (+ 1686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:31.426770 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:31.426780 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:31.426917 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:01:31.427623 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:31.427644 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:31.429034 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:31.429041 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:31.431227 (+ 2186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:31.431245 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:31.432461 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:31.432474 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:31.432855 (+ 381us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:31.433714 (+ 859us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:01:31.433717 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:01:31.433763 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:01:31.433909 (+ 146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:01:31.434379 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:01:31.434390 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:01:31.434399 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 15:01:31.434575 (+ 176us) client_negotiation.cc:240] Negotiation successful
0928 15:01:31.434971 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":341,"threads_started":1}
I20260928 15:01:31.437994 19855 heartbeater.cc:347] Connected to a master server at 127.31.40.62:45727
I20260928 15:01:31.438364 19855 heartbeater.cc:464] Registering TS with master...
I20260928 15:01:31.439191 19855 heartbeater.cc:511] Master 127.31.40.62:45727 requested a full tablet report, sending...
I20260928 15:01:31.440994 19408 ts_manager.cc:194] Registered new tserver with Master: 512260d042bd45d8885ecff5798f3809 (127.31.40.3:38441)
I20260928 15:01:31.441934 19408 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.40.3:50205
I20260928 15:01:31.445422 31904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:01:31.456956 19596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:31.449907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40582 (local address 127.31.40.62:45727)
0928 15:01:31.450208 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:31.450213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:31.450618 (+ 405us) server_negotiation.cc:407] Connection header received
0928 15:01:31.450812 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:31.450817 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:31.450902 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:31.451001 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:31.451514 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:31.452400 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:31.454072 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:31.454681 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:31.454806 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:31.456151 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:31.456162 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:31.456165 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:01:31.456191 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:31.456258 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:01:31.456274 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:31.456354 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:31.456576 (+ 222us) server_negotiation.cc:299] Negotiation successful
0928 15:01:31.456671 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 15:01:31.458108 19594 heartbeater.cc:503] Master 127.31.40.62:45727 was elected leader, sending a full tablet report...
I20260928 15:01:31.461694 19408 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40582:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:01:31.469594 19408 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:01:31.495684 19868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:31.482692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.1:42215 (local address 127.0.0.1:60690)
0928 15:01:31.484763 (+ 2071us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:31.484811 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:31.485116 (+ 305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:31.486363 (+ 1247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:31.486373 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:31.486405 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:31.486898 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:31.486918 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:31.489153 (+ 2235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:31.489160 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:31.492287 (+ 3127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:31.492304 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:31.494558 (+ 2254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:31.494566 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:31.494796 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:31.494873 (+ 77us) client_negotiation.cc:769] Sending connection context
0928 15:01:31.494990 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 15:01:31.495221 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1765,"mutex_wait_us":146,"thread_start_us":973,"threads_started":1}
I20260928 15:01:31.496332 19869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:31.484102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49118 (local address 127.31.40.3:38441)
0928 15:01:31.484966 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:31.484977 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:31.485607 (+ 630us) server_negotiation.cc:407] Connection header received
0928 15:01:31.485942 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:31.485950 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:31.486212 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:31.486405 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:31.487453 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:31.489044 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:31.493362 (+ 4318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:31.494856 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:31.495028 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:31.495351 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:31.495480 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 15:01:31.495659 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":415,"threads_started":1}
I20260928 15:01:31.496716 19871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:31.485899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.2:37817 (local address 127.0.0.1:37316)
0928 15:01:31.488024 (+ 2125us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:31.488045 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:31.488169 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:31.488774 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:31.488779 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:31.488805 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:31.489265 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:31.489281 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:31.491333 (+ 2052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:31.491338 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:31.493454 (+ 2116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:31.493474 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:31.495832 (+ 2358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:31.495839 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:31.496049 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:31.496088 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 15:01:31.496179 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 15:01:31.496344 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1989,"mutex_wait_us":1,"thread_start_us":1810,"threads_started":1}
I20260928 15:01:31.497085 19872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:31.486414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37316 (local address 127.31.40.2:37817)
0928 15:01:31.487532 (+ 1118us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:31.487544 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:31.488155 (+ 611us) server_negotiation.cc:407] Connection header received
0928 15:01:31.488340 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:31.488347 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:31.488571 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:31.488758 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:31.489537 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:31.491161 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:31.493702 (+ 2541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:31.495635 (+ 1933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:31.495943 (+ 308us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:31.496226 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:31.496309 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 15:01:31.496613 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":946,"thread_start_us":850,"threads_started":1}
I20260928 15:01:31.497313 19870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:31.483726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.3:38441 (local address 127.0.0.1:49118)
0928 15:01:31.485483 (+ 1757us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:31.485502 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:31.485725 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:31.486751 (+ 1026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:31.486758 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:31.486792 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:31.487240 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:31.487255 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:31.489244 (+ 1989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:31.489254 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:31.493152 (+ 3898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:31.493165 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:31.495071 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:31.495078 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:31.495274 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:31.495317 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 15:01:31.495423 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 15:01:31.495570 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1610,"mutex_wait_us":200,"thread_start_us":1417,"threads_started":1}
I20260928 15:01:31.497319 19867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:31.482785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60690 (local address 127.31.40.1:42215)
0928 15:01:31.483481 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:31.483491 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:31.484929 (+ 1438us) server_negotiation.cc:407] Connection header received
0928 15:01:31.485450 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:31.485462 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:31.485747 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:31.485942 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:31.487131 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:31.489005 (+ 1874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:31.492517 (+ 3512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:31.494325 (+ 1808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:31.496310 (+ 1985us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:31.496611 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:31.496693 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 15:01:31.496903 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":320,"threads_started":1}
I20260928 15:01:31.501339 19660 tablet_service.cc:1467] Processing CreateTablet for tablet 788ff29f246247a8804ca7c9a04c2e36 (DEFAULT_TABLE table=test-table [id=7152b216c61d4e2eb1e61b4dc30d6871]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:01:31.501339 19529 tablet_service.cc:1467] Processing CreateTablet for tablet 788ff29f246247a8804ca7c9a04c2e36 (DEFAULT_TABLE table=test-table [id=7152b216c61d4e2eb1e61b4dc30d6871]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:01:31.501235 19790 tablet_service.cc:1467] Processing CreateTablet for tablet 788ff29f246247a8804ca7c9a04c2e36 (DEFAULT_TABLE table=test-table [id=7152b216c61d4e2eb1e61b4dc30d6871]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:01:31.503012 19790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 788ff29f246247a8804ca7c9a04c2e36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:31.503029 19529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 788ff29f246247a8804ca7c9a04c2e36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:31.503031 19660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 788ff29f246247a8804ca7c9a04c2e36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:31.517527 19873 tablet_bootstrap.cc:492] T 788ff29f246247a8804ca7c9a04c2e36 P 512260d042bd45d8885ecff5798f3809: Bootstrap starting.
I20260928 15:01:31.519105 19875 tablet_bootstrap.cc:492] T 788ff29f246247a8804ca7c9a04c2e36 P bfc8697373634f65bc7ea8b04c8fea08: Bootstrap starting.
I20260928 15:01:31.520045 19874 tablet_bootstrap.cc:492] T 788ff29f246247a8804ca7c9a04c2e36 P 8f6c2a3592a84c49ae47fde726dcfc26: Bootstrap starting.
I20260928 15:01:31.521674 19873 tablet_bootstrap.cc:654] T 788ff29f246247a8804ca7c9a04c2e36 P 512260d042bd45d8885ecff5798f3809: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:31.523229 19873 log.cc:824] T 788ff29f246247a8804ca7c9a04c2e36 P 512260d042bd45d8885ecff5798f3809: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:31.523311 19875 tablet_bootstrap.cc:654] T 788ff29f246247a8804ca7c9a04c2e36 P bfc8697373634f65bc7ea8b04c8fea08: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:31.523959 19874 tablet_bootstrap.cc:654] T 788ff29f246247a8804ca7c9a04c2e36 P 8f6c2a3592a84c49ae47fde726dcfc26: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:31.524561 19875 log.cc:824] T 788ff29f246247a8804ca7c9a04c2e36 P bfc8697373634f65bc7ea8b04c8fea08: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:31.525177 19874 log.cc:824] T 788ff29f246247a8804ca7c9a04c2e36 P 8f6c2a3592a84c49ae47fde726dcfc26: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:31.527120 19873 tablet_bootstrap.cc:492] T 788ff29f246247a8804ca7c9a04c2e36 P 512260d042bd45d8885ecff5798f3809: No bootstrap required, opened a new log
I20260928 15:01:31.527421 19873 ts_tablet_manager.cc:1397] T 788ff29f246247a8804ca7c9a04c2e36 P 512260d042bd45d8885ecff5798f3809: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20260928 15:01:31.528633 19874 tablet_bootstrap.cc:492] T 788ff29f246247a8804ca7c9a04c2e36 P 8f6c2a3592a84c49ae47fde726dcfc26: No bootstrap required, opened a new log
I20260928 15:01:31.528952 19874 ts_tablet_manager.cc:1397] T 788ff29f246247a8804ca7c9a04c2e36 P 8f6c2a3592a84c49ae47fde726dcfc26: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.001s
I20260928 15:01:31.529500 19875 tablet_bootstrap.cc:492] T 788ff29f246247a8804ca7c9a04c2e36 P bfc8697373634f65bc7ea8b04c8fea08: No bootstrap required, opened a new log
I20260928 15:01:31.529769 19875 ts_tablet_manager.cc:1397] T 788ff29f246247a8804ca7c9a04c2e36 P bfc8697373634f65bc7ea8b04c8fea08: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260928 15:01:31.537782 19874 raft_consensus.cc:348] T 788ff29f246247a8804ca7c9a04c2e36 P 8f6c2a3592a84c49ae47fde726dcfc26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f6c2a3592a84c49ae47fde726dcfc26" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 42215 } } peers { permanent_uuid: "512260d042bd45d8885ecff5798f3809" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 38441 } } peers { permanent_uuid: "bfc8697373634f65bc7ea8b04c8fea08" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 37817 } }
I20260928 15:01:31.538322 19874 raft_consensus.cc:374] T 788ff29f246247a8804ca7c9a04c2e36 P 8f6c2a3592a84c49ae47fde726dcfc26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:31.538439 19874 raft_consensus.cc:729] T 788ff29f246247a8804ca7c9a04c2e36 P 8f6c2a3592a84c49ae47fde726dcfc26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f6c2a3592a84c49ae47fde726dcfc26, State: Initialized, Role: FOLLOWER
I20260928 15:01:31.538530 19873 raft_consensus.cc:348] T 788ff29f246247a8804ca7c9a04c2e36 P 512260d042bd45d8885ecff5798f3809 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f6c2a3592a84c49ae47fde726dcfc26" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 42215 } } peers { permanent_uuid: "512260d042bd45d8885ecff5798f3809" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 38441 } } peers { permanent_uuid: "bfc8697373634f65bc7ea8b04c8fea08" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 37817 } }
I20260928 15:01:31.539000 19873 raft_consensus.cc:374] T 788ff29f246247a8804ca7c9a04c2e36 P 512260d042bd45d8885ecff5798f3809 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:31.539100 19873 raft_consensus.cc:729] T 788ff29f246247a8804ca7c9a04c2e36 P 512260d042bd45d8885ecff5798f3809 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 512260d042bd45d8885ecff5798f3809, State: Initialized, Role: FOLLOWER
I20260928 15:01:31.539367 19874 consensus_queue.cc:260] T 788ff29f246247a8804ca7c9a04c2e36 P 8f6c2a3592a84c49ae47fde726dcfc26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f6c2a3592a84c49ae47fde726dcfc26" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 42215 } } peers { permanent_uuid: "512260d042bd45d8885ecff5798f3809" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 38441 } } peers { permanent_uuid: "bfc8697373634f65bc7ea8b04c8fea08" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 37817 } }
I20260928 15:01:31.539698 19873 consensus_queue.cc:260] T 788ff29f246247a8804ca7c9a04c2e36 P 512260d042bd45d8885ecff5798f3809 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f6c2a3592a84c49ae47fde726dcfc26" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 42215 } } peers { permanent_uuid: "512260d042bd45d8885ecff5798f3809" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 38441 } } peers { permanent_uuid: "bfc8697373634f65bc7ea8b04c8fea08" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 37817 } }
I20260928 15:01:31.540647 19875 raft_consensus.cc:348] T 788ff29f246247a8804ca7c9a04c2e36 P bfc8697373634f65bc7ea8b04c8fea08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f6c2a3592a84c49ae47fde726dcfc26" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 42215 } } peers { permanent_uuid: "512260d042bd45d8885ecff5798f3809" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 38441 } } peers { permanent_uuid: "bfc8697373634f65bc7ea8b04c8fea08" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 37817 } }
I20260928 15:01:31.541191 19875 raft_consensus.cc:374] T 788ff29f246247a8804ca7c9a04c2e36 P bfc8697373634f65bc7ea8b04c8fea08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:31.541311 19875 raft_consensus.cc:729] T 788ff29f246247a8804ca7c9a04c2e36 P bfc8697373634f65bc7ea8b04c8fea08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfc8697373634f65bc7ea8b04c8fea08, State: Initialized, Role: FOLLOWER
I20260928 15:01:31.541842 19874 ts_tablet_manager.cc:1428] T 788ff29f246247a8804ca7c9a04c2e36 P 8f6c2a3592a84c49ae47fde726dcfc26: Time spent starting tablet: real 0.013s user 0.008s sys 0.005s
I20260928 15:01:31.542049 19875 consensus_queue.cc:260] T 788ff29f246247a8804ca7c9a04c2e36 P bfc8697373634f65bc7ea8b04c8fea08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f6c2a3592a84c49ae47fde726dcfc26" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 42215 } } peers { permanent_uuid: "512260d042bd45d8885ecff5798f3809" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 38441 } } peers { permanent_uuid: "bfc8697373634f65bc7ea8b04c8fea08" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 37817 } }
I20260928 15:01:31.542860 19855 heartbeater.cc:503] Master 127.31.40.62:45727 was elected leader, sending a full tablet report...
I20260928 15:01:31.543766 19873 ts_tablet_manager.cc:1428] T 788ff29f246247a8804ca7c9a04c2e36 P 512260d042bd45d8885ecff5798f3809: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260928 15:01:31.545419 19725 heartbeater.cc:503] Master 127.31.40.62:45727 was elected leader, sending a full tablet report...
I20260928 15:01:31.546006 19875 ts_tablet_manager.cc:1428] T 788ff29f246247a8804ca7c9a04c2e36 P bfc8697373634f65bc7ea8b04c8fea08: Time spent starting tablet: real 0.016s user 0.005s sys 0.012s
W20260928 15:01:31.667855 19726 tablet.cc:2369] T 788ff29f246247a8804ca7c9a04c2e36 P bfc8697373634f65bc7ea8b04c8fea08: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:01:31.671270 19856 tablet.cc:2369] T 788ff29f246247a8804ca7c9a04c2e36 P 512260d042bd45d8885ecff5798f3809: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:01:31.675830 19879 raft_consensus.cc:482] T 788ff29f246247a8804ca7c9a04c2e36 P 8f6c2a3592a84c49ae47fde726dcfc26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:01:31.676297 19879 raft_consensus.cc:504] T 788ff29f246247a8804ca7c9a04c2e36 P 8f6c2a3592a84c49ae47fde726dcfc26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f6c2a3592a84c49ae47fde726dcfc26" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 42215 } } peers { permanent_uuid: "512260d042bd45d8885ecff5798f3809" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 38441 } } peers { permanent_uuid: "bfc8697373634f65bc7ea8b04c8fea08" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 37817 } }
I20260928 15:01:31.678493 19879 leader_election.cc:288] T 788ff29f246247a8804ca7c9a04c2e36 P 8f6c2a3592a84c49ae47fde726dcfc26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 512260d042bd45d8885ecff5798f3809 (127.31.40.3:38441), bfc8697373634f65bc7ea8b04c8fea08 (127.31.40.2:37817)
W20260928 15:01:31.686081 19595 tablet.cc:2369] T 788ff29f246247a8804ca7c9a04c2e36 P 8f6c2a3592a84c49ae47fde726dcfc26: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:01:31.686264 19869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:31.679000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:57381 (local address 127.31.40.3:38441)
0928 15:01:31.679210 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:31.679218 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:31.679895 (+ 677us) server_negotiation.cc:407] Connection header received
0928 15:01:31.680129 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:31.680135 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:31.680218 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:31.680331 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:31.681066 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:31.682004 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:31.684297 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:31.685186 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:31.685297 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:31.685377 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:31.685864 (+ 487us) server_negotiation.cc:299] Negotiation successful
0928 15:01:31.686005 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 15:01:31.686558 19883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:31.678980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.2:37817 (local address 127.31.40.1:49865)
0928 15:01:31.680409 (+ 1429us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:31.680425 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:31.680554 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:31.680937 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:31.680984 (+ 47us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:31.681010 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:31.681407 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:31.681421 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:31.682800 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:31.682806 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:31.684368 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:31.684382 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:31.685770 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:31.685778 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:31.685938 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:31.685971 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 15:01:31.686069 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 15:01:31.686198 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1308,"thread_start_us":1088,"threads_started":1}
I20260928 15:01:31.686640 19872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:31.679115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.40.1:49865 (local address 127.31.40.2:37817)
0928 15:01:31.679428 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:31.679436 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:31.680505 (+ 1069us) server_negotiation.cc:407] Connection header received
0928 15:01:31.680702 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:31.680707 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:31.680812 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:31.680921 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:31.681592 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:31.682628 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:31.684576 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:31.685526 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:31.685672 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:31.685785 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:31.686124 (+ 339us) server_negotiation.cc:299] Negotiation successful
0928 15:01:31.686287 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161}
I20260928 15:01:31.687147 19882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:31.678785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.3:38441 (local address 127.31.40.1:57381)
0928 15:01:31.679705 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:31.679723 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:31.679957 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:31.680356 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:31.680362 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:31.680407 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:31.680874 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:31.680891 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:31.682154 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:31.682160 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:31.684104 (+ 1944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:31.684118 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:31.685399 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:31.685408 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:31.685591 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:31.685639 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 15:01:31.685727 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 15:01:31.686824 (+ 1097us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":793,"mutex_wait_us":11,"thread_start_us":551,"threads_started":1}
I20260928 15:01:31.687577 19680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "788ff29f246247a8804ca7c9a04c2e36" candidate_uuid: "8f6c2a3592a84c49ae47fde726dcfc26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfc8697373634f65bc7ea8b04c8fea08" is_pre_election: true
I20260928 15:01:31.688266 19680 raft_consensus.cc:2454] T 788ff29f246247a8804ca7c9a04c2e36 P bfc8697373634f65bc7ea8b04c8fea08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f6c2a3592a84c49ae47fde726dcfc26 in term 0.
I20260928 15:01:31.688226 19810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "788ff29f246247a8804ca7c9a04c2e36" candidate_uuid: "8f6c2a3592a84c49ae47fde726dcfc26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "512260d042bd45d8885ecff5798f3809" is_pre_election: true
I20260928 15:01:31.688761 19810 raft_consensus.cc:2454] T 788ff29f246247a8804ca7c9a04c2e36 P 512260d042bd45d8885ecff5798f3809 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f6c2a3592a84c49ae47fde726dcfc26 in term 0.
I20260928 15:01:31.689037 19482 leader_election.cc:302] T 788ff29f246247a8804ca7c9a04c2e36 P 8f6c2a3592a84c49ae47fde726dcfc26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f6c2a3592a84c49ae47fde726dcfc26, bfc8697373634f65bc7ea8b04c8fea08; no voters:
I20260928 15:01:31.689502 19879 raft_consensus.cc:2781] T 788ff29f246247a8804ca7c9a04c2e36 P 8f6c2a3592a84c49ae47fde726dcfc26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:01:31.689625 19879 raft_consensus.cc:482] T 788ff29f246247a8804ca7c9a04c2e36 P 8f6c2a3592a84c49ae47fde726dcfc26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:01:31.689765 19879 raft_consensus.cc:3037] T 788ff29f246247a8804ca7c9a04c2e36 P 8f6c2a3592a84c49ae47fde726dcfc26 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:31.692168 19879 raft_consensus.cc:504] T 788ff29f246247a8804ca7c9a04c2e36 P 8f6c2a3592a84c49ae47fde726dcfc26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f6c2a3592a84c49ae47fde726dcfc26" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 42215 } } peers { permanent_uuid: "512260d042bd45d8885ecff5798f3809" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 38441 } } peers { permanent_uuid: "bfc8697373634f65bc7ea8b04c8fea08" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 37817 } }
I20260928 15:01:31.692889 19879 leader_election.cc:288] T 788ff29f246247a8804ca7c9a04c2e36 P 8f6c2a3592a84c49ae47fde726dcfc26 [CANDIDATE]: Term 1 election: Requested vote from peers 512260d042bd45d8885ecff5798f3809 (127.31.40.3:38441), bfc8697373634f65bc7ea8b04c8fea08 (127.31.40.2:37817)
I20260928 15:01:31.693369 19810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "788ff29f246247a8804ca7c9a04c2e36" candidate_uuid: "8f6c2a3592a84c49ae47fde726dcfc26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "512260d042bd45d8885ecff5798f3809"
I20260928 15:01:31.693536 19680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "788ff29f246247a8804ca7c9a04c2e36" candidate_uuid: "8f6c2a3592a84c49ae47fde726dcfc26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfc8697373634f65bc7ea8b04c8fea08"
I20260928 15:01:31.693578 19810 raft_consensus.cc:3037] T 788ff29f246247a8804ca7c9a04c2e36 P 512260d042bd45d8885ecff5798f3809 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:31.693776 19680 raft_consensus.cc:3037] T 788ff29f246247a8804ca7c9a04c2e36 P bfc8697373634f65bc7ea8b04c8fea08 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:31.696013 19810 raft_consensus.cc:2454] T 788ff29f246247a8804ca7c9a04c2e36 P 512260d042bd45d8885ecff5798f3809 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f6c2a3592a84c49ae47fde726dcfc26 in term 1.
I20260928 15:01:31.696035 19680 raft_consensus.cc:2454] T 788ff29f246247a8804ca7c9a04c2e36 P bfc8697373634f65bc7ea8b04c8fea08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f6c2a3592a84c49ae47fde726dcfc26 in term 1.
I20260928 15:01:31.696482 19484 leader_election.cc:302] T 788ff29f246247a8804ca7c9a04c2e36 P 8f6c2a3592a84c49ae47fde726dcfc26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 512260d042bd45d8885ecff5798f3809, 8f6c2a3592a84c49ae47fde726dcfc26; no voters:
I20260928 15:01:31.696816 19879 raft_consensus.cc:2781] T 788ff29f246247a8804ca7c9a04c2e36 P 8f6c2a3592a84c49ae47fde726dcfc26 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:01:31.697346 19879 raft_consensus.cc:686] T 788ff29f246247a8804ca7c9a04c2e36 P 8f6c2a3592a84c49ae47fde726dcfc26 [term 1 LEADER]: Becoming Leader. State: Replica: 8f6c2a3592a84c49ae47fde726dcfc26, State: Running, Role: LEADER
I20260928 15:01:31.697892 19879 consensus_queue.cc:237] T 788ff29f246247a8804ca7c9a04c2e36 P 8f6c2a3592a84c49ae47fde726dcfc26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f6c2a3592a84c49ae47fde726dcfc26" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 42215 } } peers { permanent_uuid: "512260d042bd45d8885ecff5798f3809" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 38441 } } peers { permanent_uuid: "bfc8697373634f65bc7ea8b04c8fea08" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 37817 } }
I20260928 15:01:31.702630 19408 catalog_manager.cc:5495] T 788ff29f246247a8804ca7c9a04c2e36 P 8f6c2a3592a84c49ae47fde726dcfc26 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f6c2a3592a84c49ae47fde726dcfc26 (127.31.40.1). New cstate: current_term: 1 leader_uuid: "8f6c2a3592a84c49ae47fde726dcfc26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f6c2a3592a84c49ae47fde726dcfc26" member_type: VOTER last_known_addr { host: "127.31.40.1" port: 42215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "512260d042bd45d8885ecff5798f3809" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 38441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfc8697373634f65bc7ea8b04c8fea08" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 37817 } health_report { overall_health: UNKNOWN } } }
I20260928 15:01:31.737804 19867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:31.732024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60704 (local address 127.31.40.1:42215)
0928 15:01:31.732239 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:31.732244 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:31.732413 (+ 169us) server_negotiation.cc:407] Connection header received
0928 15:01:31.732537 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:31.732540 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:31.732644 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:31.732919 (+ 275us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:01:31.733388 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:31.734120 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:31.735506 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:31.735968 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:31.736111 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:31.736658 (+ 547us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:01:31.737211 (+ 553us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:01:31.737386 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:31.737445 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 15:01:31.737530 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 15:01:31.746694 19680 raft_consensus.cc:1261] T 788ff29f246247a8804ca7c9a04c2e36 P bfc8697373634f65bc7ea8b04c8fea08 [term 1 FOLLOWER]: Refusing update from remote peer 8f6c2a3592a84c49ae47fde726dcfc26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:01:31.746867 19810 raft_consensus.cc:1261] T 788ff29f246247a8804ca7c9a04c2e36 P 512260d042bd45d8885ecff5798f3809 [term 1 FOLLOWER]: Refusing update from remote peer 8f6c2a3592a84c49ae47fde726dcfc26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:01:31.747845 19884 consensus_queue.cc:1035] T 788ff29f246247a8804ca7c9a04c2e36 P 8f6c2a3592a84c49ae47fde726dcfc26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfc8697373634f65bc7ea8b04c8fea08" member_type: VOTER last_known_addr { host: "127.31.40.2" port: 37817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:01:31.748291 19879 consensus_queue.cc:1035] T 788ff29f246247a8804ca7c9a04c2e36 P 8f6c2a3592a84c49ae47fde726dcfc26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "512260d042bd45d8885ecff5798f3809" member_type: VOTER last_known_addr { host: "127.31.40.3" port: 38441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:01:31.770293 19889 mvcc.cc:205] Tried to move back new op lower bound from 7334329105384087552 to 7334329105198460928. Current Snapshot: MvccSnapshot[applied={T|T < 7334329105384087552}]
I20260928 15:01:31.777068 19408 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:01:31.777345 19408 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:01:31.781572 19408 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 32064cce24694da6be3580143a0e4b28: Sending DeleteTablet for 3 replicas of tablet 788ff29f246247a8804ca7c9a04c2e36
I20260928 15:01:31.782974 19529 tablet_service.cc:1514] Processing DeleteTablet for tablet 788ff29f246247a8804ca7c9a04c2e36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:01:31 UTC) from {username='slave'} at 127.0.0.1:60690
I20260928 15:01:31.783133 19660 tablet_service.cc:1514] Processing DeleteTablet for tablet 788ff29f246247a8804ca7c9a04c2e36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:01:31 UTC) from {username='slave'} at 127.0.0.1:37316
I20260928 15:01:31.783766 19790 tablet_service.cc:1514] Processing DeleteTablet for tablet 788ff29f246247a8804ca7c9a04c2e36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:01:31 UTC) from {username='slave'} at 127.0.0.1:49118
I20260928 15:01:31.784248 19897 tablet_replica.cc:330] T 788ff29f246247a8804ca7c9a04c2e36 P 8f6c2a3592a84c49ae47fde726dcfc26: stopping tablet replica
I20260928 15:01:31.784915 19897 raft_consensus.cc:2229] T 788ff29f246247a8804ca7c9a04c2e36 P 8f6c2a3592a84c49ae47fde726dcfc26 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:01:31.785485 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 19467
I20260928 15:01:31.785979 19897 raft_consensus.cc:2258] T 788ff29f246247a8804ca7c9a04c2e36 P 8f6c2a3592a84c49ae47fde726dcfc26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:01:31.788836 19897 ts_tablet_manager.cc:1905] T 788ff29f246247a8804ca7c9a04c2e36 P 8f6c2a3592a84c49ae47fde726dcfc26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:01:31.799993 19897 ts_tablet_manager.cc:1918] T 788ff29f246247a8804ca7c9a04c2e36 P 8f6c2a3592a84c49ae47fde726dcfc26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:01:31.800419 19897 log.cc:1196] T 788ff29f246247a8804ca7c9a04c2e36 P 8f6c2a3592a84c49ae47fde726dcfc26: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/ts-0/wal/wals/788ff29f246247a8804ca7c9a04c2e36
I20260928 15:01:31.801317 19897 ts_tablet_manager.cc:1939] T 788ff29f246247a8804ca7c9a04c2e36 P 8f6c2a3592a84c49ae47fde726dcfc26: Deleting consensus metadata
I20260928 15:01:31.804064 19393 catalog_manager.cc:4841] TS 8f6c2a3592a84c49ae47fde726dcfc26 (127.31.40.1:42215): tablet 788ff29f246247a8804ca7c9a04c2e36 (table test-table [id=7152b216c61d4e2eb1e61b4dc30d6871]) successfully deleted
I20260928 15:01:31.810823 19899 tablet_replica.cc:330] T 788ff29f246247a8804ca7c9a04c2e36 P 512260d042bd45d8885ecff5798f3809: stopping tablet replica
I20260928 15:01:31.811488 19899 raft_consensus.cc:2229] T 788ff29f246247a8804ca7c9a04c2e36 P 512260d042bd45d8885ecff5798f3809 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:01:31.812232 19899 raft_consensus.cc:2258] T 788ff29f246247a8804ca7c9a04c2e36 P 512260d042bd45d8885ecff5798f3809 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:01:31.814764 19899 ts_tablet_manager.cc:1905] T 788ff29f246247a8804ca7c9a04c2e36 P 512260d042bd45d8885ecff5798f3809: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:01:31.820206 19898 tablet_replica.cc:330] T 788ff29f246247a8804ca7c9a04c2e36 P bfc8697373634f65bc7ea8b04c8fea08: stopping tablet replica
I20260928 15:01:31.820639 19898 raft_consensus.cc:2229] T 788ff29f246247a8804ca7c9a04c2e36 P bfc8697373634f65bc7ea8b04c8fea08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:01:31.821151 19898 raft_consensus.cc:2258] T 788ff29f246247a8804ca7c9a04c2e36 P bfc8697373634f65bc7ea8b04c8fea08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:01:31.822896 19899 ts_tablet_manager.cc:1918] T 788ff29f246247a8804ca7c9a04c2e36 P 512260d042bd45d8885ecff5798f3809: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:01:31.823076 19899 log.cc:1196] T 788ff29f246247a8804ca7c9a04c2e36 P 512260d042bd45d8885ecff5798f3809: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/ts-2/wal/wals/788ff29f246247a8804ca7c9a04c2e36
I20260928 15:01:31.823644 19899 ts_tablet_manager.cc:1939] T 788ff29f246247a8804ca7c9a04c2e36 P 512260d042bd45d8885ecff5798f3809: Deleting consensus metadata
I20260928 15:01:31.824049 19867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:31.787320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60712 (local address 127.31.40.1:42215)
0928 15:01:31.787507 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:31.787513 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:31.808589 (+ 21076us) server_negotiation.cc:407] Connection header received
0928 15:01:31.809019 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:31.809028 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:31.809204 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:31.809318 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:31.812819 (+ 3501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:31.814502 (+ 1683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:31.817455 (+ 2953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:31.818403 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:31.818552 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:31.822099 (+ 3547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:31.822112 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:31.822135 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:01:31.823026 (+ 891us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:31.823099 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:01:31.823140 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:31.823285 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:31.823463 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 15:01:31.823583 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 15:01:31.824620 19898 ts_tablet_manager.cc:1905] T 788ff29f246247a8804ca7c9a04c2e36 P bfc8697373634f65bc7ea8b04c8fea08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:01:31.824995 19589 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:01:31.825910 19394 catalog_manager.cc:4841] TS 512260d042bd45d8885ecff5798f3809 (127.31.40.3:38441): tablet 788ff29f246247a8804ca7c9a04c2e36 (table test-table [id=7152b216c61d4e2eb1e61b4dc30d6871]) successfully deleted
I20260928 15:01:31.832899 19898 ts_tablet_manager.cc:1918] T 788ff29f246247a8804ca7c9a04c2e36 P bfc8697373634f65bc7ea8b04c8fea08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:01:31.833088 19898 log.cc:1196] T 788ff29f246247a8804ca7c9a04c2e36 P bfc8697373634f65bc7ea8b04c8fea08: Deleting WAL directory at /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607365362981-31904-0/minicluster-data/ts-1/wal/wals/788ff29f246247a8804ca7c9a04c2e36
I20260928 15:01:31.833575 19898 ts_tablet_manager.cc:1939] T 788ff29f246247a8804ca7c9a04c2e36 P bfc8697373634f65bc7ea8b04c8fea08: Deleting consensus metadata
I20260928 15:01:31.835369 19392 catalog_manager.cc:4841] TS bfc8697373634f65bc7ea8b04c8fea08 (127.31.40.2:37817): tablet 788ff29f246247a8804ca7c9a04c2e36 (table test-table [id=7152b216c61d4e2eb1e61b4dc30d6871]) successfully deleted
I20260928 15:01:31.927528 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 19467
I20260928 15:01:31.949713 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 19598
I20260928 15:01:31.957055 19872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:31.950596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37332 (local address 127.31.40.2:37817)
0928 15:01:31.950796 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:31.950803 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:31.950823 (+ 20us) server_negotiation.cc:407] Connection header received
0928 15:01:31.950986 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:31.950990 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:31.951068 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:31.951177 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:31.951623 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:31.952463 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:31.953888 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:31.954379 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:31.954512 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:31.955579 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:31.955588 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:31.955603 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:01:31.956283 (+ 680us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:31.956331 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:01:31.956350 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:31.956480 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:31.956609 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 15:01:31.956712 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 15:01:31.957798 19720 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:01:32.050391 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 19598
I20260928 15:01:32.068614 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 19728
I20260928 15:01:32.075996 19869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:32.069516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49130 (local address 127.31.40.3:38441)
0928 15:01:32.069674 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:32.069679 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:32.069806 (+ 127us) server_negotiation.cc:407] Connection header received
0928 15:01:32.070029 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:32.070036 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:32.070180 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:32.070289 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:32.070707 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:32.071458 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:32.072821 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:32.073216 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:32.073360 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:32.074512 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:32.074522 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:32.074539 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:01:32.075124 (+ 585us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:32.075180 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:01:32.075200 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:32.075330 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:32.075471 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 15:01:32.075586 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 15:01:32.076730 19850 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:01:32.174538 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 19728
I20260928 15:01:32.192086 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 19377
I20260928 15:01:32.193393 19438 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:01:32.303835 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 19377
2026-09-28T15:01:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2973 ms)
[----------] 6 tests from EncryptionPolicyTest (22311 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607365362981-31904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:01:32 dist-test-slave-j968 krb5kdc[19907](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:01:32 dist-test-slave-j968 krb5kdc[19907](info): set up 2 sockets
Sep 28 15:01:32 dist-test-slave-j968 krb5kdc[19907](info): commencing operation
krb5kdc: starting...
W20260928 15:01:33.525419 31904 mini_kdc.cc:121] Time spent starting KDC: real 1.170s user 0.006s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Sep 28 15:01:33 dist-test-slave-j968 krb5kdc[19907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:01:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:01:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607365362981-31904-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607365362981-31904-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:01:33.804347 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607365362981-31904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607365362981-31904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607365362981-31904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33855
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607365362981-31904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:01:34.162474 19920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:34.162811 19920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:34.162982 19920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:34.171130 19920 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:01:34.171247 19920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:34.171291 19920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:34.171376 19920 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:01:34.171445 19920 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:01:34.181912 19920 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607365362981-31904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607365362981-31904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33855
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607365362981-31904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607365362981-31904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:01:34.183393 19920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:34.184827 19920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:34.197065 19928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:34.197223 19925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:34.197129 19926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:34.197853 19920 server_base.cc:1034] running on GCE node
I20260928 15:01:34.198680 19920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:34.200222 19920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:34.201474 19920 hybrid_clock.cc:648] HybridClock initialized: now 1790607694201407 us; error 76 us; skew 500 ppm
I20260928 15:01:34.201867 19920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:01:34 dist-test-slave-j968 krb5kdc[19907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:01:34.205171 19920 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:01:34.206638 19920 webserver.cc:460] Webserver started at http://127.0.0.1:45517/ using document root <none> and password file <none>
I20260928 15:01:34.207290 19920 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:34.207433 19920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:34.207706 19920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:34.210807 19920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607365362981-31904-0/minicluster-data/master-0/data/instance:
uuid: "d375fc7ac1c046c6acbd5c5ca42aa07d"
format_stamp: "Formatted at 2026-09-28 15:01:34 on dist-test-slave-j968"
server_key: "86a26a3bc1462efaba9a241d24860697"
server_key_iv: "c86b26ac98ad78ecbf982f95aafe822e"
server_key_version: "encryptionkey@0"
I20260928 15:01:34.211545 19920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607365362981-31904-0/minicluster-data/master-0/wal/instance:
uuid: "d375fc7ac1c046c6acbd5c5ca42aa07d"
format_stamp: "Formatted at 2026-09-28 15:01:34 on dist-test-slave-j968"
server_key: "86a26a3bc1462efaba9a241d24860697"
server_key_iv: "c86b26ac98ad78ecbf982f95aafe822e"
server_key_version: "encryptionkey@0"
I20260928 15:01:34.217018 19920 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260928 15:01:34.220914 19935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:34.222442 19920 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:01:34.222661 19920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607365362981-31904-0/minicluster-data/master-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607365362981-31904-0/minicluster-data/master-0/wal
uuid: "d375fc7ac1c046c6acbd5c5ca42aa07d"
format_stamp: "Formatted at 2026-09-28 15:01:34 on dist-test-slave-j968"
server_key: "86a26a3bc1462efaba9a241d24860697"
server_key_iv: "c86b26ac98ad78ecbf982f95aafe822e"
server_key_version: "encryptionkey@0"
I20260928 15:01:34.222898 19920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607365362981-31904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607365362981-31904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607365362981-31904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:34.242786 19920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:34.247321 19920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:34.247634 19920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:34.267457 19920 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33855
I20260928 15:01:34.267490 19986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33855 every 8 connection(s)
I20260928 15:01:34.268934 19920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
I20260928 15:01:34.271617 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 19920
I20260928 15:01:34.271847 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607365362981-31904-0/minicluster-data/master-0/wal/instance
I20260928 15:01:34.272657 31904 external_mini_cluster.cc:1428] Setting key ac884011eb6c04d090b00e370eac2cbd
I20260928 15:01:34.273638 19987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 15:01:34 dist-test-slave-j968 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 15:01:34.288782 19987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d375fc7ac1c046c6acbd5c5ca42aa07d: Bootstrap starting.
I20260928 15:01:34.292374 19987 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d375fc7ac1c046c6acbd5c5ca42aa07d: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:34.293418 19987 log.cc:824] T 00000000000000000000000000000000 P d375fc7ac1c046c6acbd5c5ca42aa07d: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:34.296507 19987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d375fc7ac1c046c6acbd5c5ca42aa07d: No bootstrap required, opened a new log
I20260928 15:01:34.296648 19990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:34.274650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44951 (local address 127.0.0.1:33855)
0928 15:01:34.275464 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:34.275480 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:34.275552 (+ 72us) server_negotiation.cc:407] Connection header received
0928 15:01:34.277250 (+ 1698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:34.277273 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:34.277683 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:34.278030 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:34.279966 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:34.281327 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:34.283120 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:34.283784 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:34.284046 (+ 262us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:34.288946 (+ 4900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:34.288957 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:34.288978 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:34.289034 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:34.292133 (+ 3099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:34.293091 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:34.293098 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:34.293104 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:34.293273 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:34.293818 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:34.293829 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:34.293832 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:34.294397 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:34.294749 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:34.295240 (+ 491us) server_negotiation.cc:299] Negotiation successful
0928 15:01:34.295567 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":365,"threads_started":1}
I20260928 15:01:34.301010 31904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:01:34.305464 19987 raft_consensus.cc:348] T 00000000000000000000000000000000 P d375fc7ac1c046c6acbd5c5ca42aa07d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d375fc7ac1c046c6acbd5c5ca42aa07d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33855 } }
I20260928 15:01:34.305855 19987 raft_consensus.cc:374] T 00000000000000000000000000000000 P d375fc7ac1c046c6acbd5c5ca42aa07d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:34.305959 19987 raft_consensus.cc:729] T 00000000000000000000000000000000 P d375fc7ac1c046c6acbd5c5ca42aa07d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d375fc7ac1c046c6acbd5c5ca42aa07d, State: Initialized, Role: FOLLOWER
I20260928 15:01:34.306738 19987 consensus_queue.cc:260] T 00000000000000000000000000000000 P d375fc7ac1c046c6acbd5c5ca42aa07d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d375fc7ac1c046c6acbd5c5ca42aa07d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33855 } }
I20260928 15:01:34.307034 19987 raft_consensus.cc:388] T 00000000000000000000000000000000 P d375fc7ac1c046c6acbd5c5ca42aa07d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:01:34.307190 19987 raft_consensus.cc:482] T 00000000000000000000000000000000 P d375fc7ac1c046c6acbd5c5ca42aa07d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:01:34.307454 19987 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d375fc7ac1c046c6acbd5c5ca42aa07d [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:34.310636 19987 raft_consensus.cc:504] T 00000000000000000000000000000000 P d375fc7ac1c046c6acbd5c5ca42aa07d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d375fc7ac1c046c6acbd5c5ca42aa07d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33855 } }
I20260928 15:01:34.311298 19987 leader_election.cc:302] T 00000000000000000000000000000000 P d375fc7ac1c046c6acbd5c5ca42aa07d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d375fc7ac1c046c6acbd5c5ca42aa07d; no voters:
I20260928 15:01:34.312063 19987 leader_election.cc:288] T 00000000000000000000000000000000 P d375fc7ac1c046c6acbd5c5ca42aa07d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:01:34.312179 19997 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d375fc7ac1c046c6acbd5c5ca42aa07d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:01:34.314291 19987 sys_catalog.cc:564] T 00000000000000000000000000000000 P d375fc7ac1c046c6acbd5c5ca42aa07d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:01:34.314894 19997 raft_consensus.cc:686] T 00000000000000000000000000000000 P d375fc7ac1c046c6acbd5c5ca42aa07d [term 1 LEADER]: Becoming Leader. State: Replica: d375fc7ac1c046c6acbd5c5ca42aa07d, State: Running, Role: LEADER
I20260928 15:01:34.315584 19997 consensus_queue.cc:237] T 00000000000000000000000000000000 P d375fc7ac1c046c6acbd5c5ca42aa07d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d375fc7ac1c046c6acbd5c5ca42aa07d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33855 } }
I20260928 15:01:34.320919 19998 sys_catalog.cc:455] T 00000000000000000000000000000000 P d375fc7ac1c046c6acbd5c5ca42aa07d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d375fc7ac1c046c6acbd5c5ca42aa07d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d375fc7ac1c046c6acbd5c5ca42aa07d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33855 } } }
I20260928 15:01:34.322108 19998 sys_catalog.cc:458] T 00000000000000000000000000000000 P d375fc7ac1c046c6acbd5c5ca42aa07d [sys.catalog]: This master's current role is: LEADER
I20260928 15:01:34.320725 19999 sys_catalog.cc:455] T 00000000000000000000000000000000 P d375fc7ac1c046c6acbd5c5ca42aa07d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d375fc7ac1c046c6acbd5c5ca42aa07d. Latest consensus state: current_term: 1 leader_uuid: "d375fc7ac1c046c6acbd5c5ca42aa07d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d375fc7ac1c046c6acbd5c5ca42aa07d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33855 } } }
I20260928 15:01:34.322436 19999 sys_catalog.cc:458] T 00000000000000000000000000000000 P d375fc7ac1c046c6acbd5c5ca42aa07d [sys.catalog]: This master's current role is: LEADER
I20260928 15:01:34.328872 19990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:34.306309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33591 (local address 127.0.0.1:33855)
0928 15:01:34.306555 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:34.306564 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:34.307184 (+ 620us) server_negotiation.cc:407] Connection header received
0928 15:01:34.307311 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:34.307319 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:34.307410 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:34.307498 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:34.309552 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:34.310333 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:34.312331 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:34.313331 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:34.313495 (+ 164us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:34.317337 (+ 3842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:34.317344 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:34.317349 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:34.317396 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:34.322917 (+ 5521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:34.324481 (+ 1564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:34.324490 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:34.324496 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:34.324733 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:34.325484 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:34.325492 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:34.325498 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:34.325986 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:34.327789 (+ 1803us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:34.328031 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 15:01:34.328286 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 15:01:34.333220 20010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:01:34.338711 20010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:01:34.347512 20010 catalog_manager.cc:1347] Generated new cluster ID: e7afab3fd9ba4629b5702795df9e8d6f
I20260928 15:01:34.347630 20010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:01:34.365068 20010 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:01:34.366752 20010 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:01:34.376940 20010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d375fc7ac1c046c6acbd5c5ca42aa07d: Generated new TSK 0
I20260928 15:01:34.377996 20010 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:01:34.392817 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 19920
I20260928 15:01:34.394169 19982 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:01:34.492558 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 19920
W20260928 15:01:34.511140 19905 connection.cc:535] client connection to 127.0.0.1:33855 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33855): Connection reset by peer (error 104)
2026-09-28T15:01:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2175 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607365362981-31904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:01:34 dist-test-slave-j968 krb5kdc[20019](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:01:34 dist-test-slave-j968 krb5kdc[20019](info): set up 2 sockets
Sep 28 15:01:34 dist-test-slave-j968 krb5kdc[20019](info): commencing operation
krb5kdc: starting...
W20260928 15:01:35.629290 31904 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Sep 28 15:01:35 dist-test-slave-j968 krb5kdc[20019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:01:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:01:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607365362981-31904-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607365362981-31904-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:01:35.900185 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607365362981-31904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607365362981-31904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607365362981-31904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40613
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607365362981-31904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40613
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:01:36.257232 20032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:36.257598 20032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:36.257762 20032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:36.265120 20032 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:01:36.265281 20032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:36.265377 20032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:36.265504 20032 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:01:36.265581 20032 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:01:36.275508 20032 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607365362981-31904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607365362981-31904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40613
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607365362981-31904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607365362981-31904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:01:36.276913 20032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:36.278239 20032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:36.290601 20037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:36.290503 20038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:36.290571 20040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:36.291455 20032 server_base.cc:1034] running on GCE node
I20260928 15:01:36.292337 20032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:36.293795 20032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:36.295032 20032 hybrid_clock.cc:648] HybridClock initialized: now 1790607696294970 us; error 46 us; skew 500 ppm
I20260928 15:01:36.295395 20032 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:01:36 dist-test-slave-j968 krb5kdc[20019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:01:36.298695 20032 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:01:36.300354 20032 webserver.cc:460] Webserver started at http://127.0.0.1:36661/ using document root <none> and password file <none>
I20260928 15:01:36.301057 20032 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:36.301205 20032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:36.301527 20032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:36.304693 20032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607365362981-31904-0/minicluster-data/master-0/data/instance:
uuid: "a8d07999846d46a8b293bdb7ba550321"
format_stamp: "Formatted at 2026-09-28 15:01:36 on dist-test-slave-j968"
server_key: "a1cf2b8960d97eb323f5872680ad2e75"
server_key_iv: "2fe0c613d02b7c21070c68253dcf134b"
server_key_version: "encryptionkey@0"
I20260928 15:01:36.305501 20032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607365362981-31904-0/minicluster-data/master-0/wal/instance:
uuid: "a8d07999846d46a8b293bdb7ba550321"
format_stamp: "Formatted at 2026-09-28 15:01:36 on dist-test-slave-j968"
server_key: "a1cf2b8960d97eb323f5872680ad2e75"
server_key_iv: "2fe0c613d02b7c21070c68253dcf134b"
server_key_version: "encryptionkey@0"
I20260928 15:01:36.310844 20032 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 15:01:36.314572 20047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:36.316133 20032 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:01:36.316334 20032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607365362981-31904-0/minicluster-data/master-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607365362981-31904-0/minicluster-data/master-0/wal
uuid: "a8d07999846d46a8b293bdb7ba550321"
format_stamp: "Formatted at 2026-09-28 15:01:36 on dist-test-slave-j968"
server_key: "a1cf2b8960d97eb323f5872680ad2e75"
server_key_iv: "2fe0c613d02b7c21070c68253dcf134b"
server_key_version: "encryptionkey@0"
I20260928 15:01:36.316563 20032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607365362981-31904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607365362981-31904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607365362981-31904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:36.335950 20032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:36.340529 20032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:36.340842 20032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:36.360693 20032 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40613
I20260928 15:01:36.360733 20098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40613 every 8 connection(s)
I20260928 15:01:36.362150 20032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
I20260928 15:01:36.363636 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 20032
I20260928 15:01:36.363831 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607365362981-31904-0/minicluster-data/master-0/wal/instance
I20260928 15:01:36.364326 31904 external_mini_cluster.cc:1428] Setting key 8be501a34af3549909dfad0caa87045f
I20260928 15:01:36.366976 20099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:36.380981 20102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:36.366240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37249 (local address 127.0.0.1:40613)
0928 15:01:36.367126 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:36.367142 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:36.367196 (+ 54us) server_negotiation.cc:407] Connection header received
0928 15:01:36.369734 (+ 2538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:36.369773 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:36.370358 (+ 585us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:36.370862 (+ 504us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:36.371447 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:36.373782 (+ 2335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:36.375124 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:36.376332 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:36.376518 (+ 186us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:36.377790 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:36.377802 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:36.377827 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:01:36.378639 (+ 812us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:36.378747 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:01:36.378797 (+ 50us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:36.378970 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:36.379223 (+ 253us) server_negotiation.cc:299] Negotiation successful
0928 15:01:36.379424 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":340,"threads_started":1}
I20260928 15:01:36.382623 20099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8d07999846d46a8b293bdb7ba550321: Bootstrap starting.
I20260928 15:01:36.384923 31904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:01:36.386413 20099 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8d07999846d46a8b293bdb7ba550321: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:36.387782 20099 log.cc:824] T 00000000000000000000000000000000 P a8d07999846d46a8b293bdb7ba550321: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:36.391415 20099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8d07999846d46a8b293bdb7ba550321: No bootstrap required, opened a new log
I20260928 15:01:36.397967 20102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:36.390773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60719 (local address 127.0.0.1:40613)
0928 15:01:36.390958 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:36.390965 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:36.391262 (+ 297us) server_negotiation.cc:407] Connection header received
0928 15:01:36.391425 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:36.391433 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:36.391520 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:36.391637 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:36.392179 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:36.393550 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:36.394908 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:36.395681 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:36.395838 (+ 157us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:36.397026 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:36.397035 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:36.397043 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:01:36.397071 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:36.397145 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:01:36.397167 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:36.397302 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:36.397511 (+ 209us) server_negotiation.cc:299] Negotiation successful
0928 15:01:36.397581 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 15:01:36.402230 20099 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8d07999846d46a8b293bdb7ba550321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8d07999846d46a8b293bdb7ba550321" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40613 } }
I20260928 15:01:36.402648 20099 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8d07999846d46a8b293bdb7ba550321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:36.402729 20099 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8d07999846d46a8b293bdb7ba550321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8d07999846d46a8b293bdb7ba550321, State: Initialized, Role: FOLLOWER
I20260928 15:01:36.403296 20099 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8d07999846d46a8b293bdb7ba550321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8d07999846d46a8b293bdb7ba550321" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40613 } }
I20260928 15:01:36.403519 20099 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8d07999846d46a8b293bdb7ba550321 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:01:36.403644 20099 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8d07999846d46a8b293bdb7ba550321 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:01:36.403788 20099 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8d07999846d46a8b293bdb7ba550321 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:36.406307 20099 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8d07999846d46a8b293bdb7ba550321 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8d07999846d46a8b293bdb7ba550321" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40613 } }
I20260928 15:01:36.406860 20099 leader_election.cc:302] T 00000000000000000000000000000000 P a8d07999846d46a8b293bdb7ba550321 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8d07999846d46a8b293bdb7ba550321; no voters:
I20260928 15:01:36.407480 20099 leader_election.cc:288] T 00000000000000000000000000000000 P a8d07999846d46a8b293bdb7ba550321 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:01:36.407671 20109 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8d07999846d46a8b293bdb7ba550321 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:01:36.409210 20109 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8d07999846d46a8b293bdb7ba550321 [term 1 LEADER]: Becoming Leader. State: Replica: a8d07999846d46a8b293bdb7ba550321, State: Running, Role: LEADER
I20260928 15:01:36.409960 20109 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8d07999846d46a8b293bdb7ba550321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8d07999846d46a8b293bdb7ba550321" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40613 } }
I20260928 15:01:36.410209 20099 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8d07999846d46a8b293bdb7ba550321 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:01:36.413323 20111 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8d07999846d46a8b293bdb7ba550321 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8d07999846d46a8b293bdb7ba550321. Latest consensus state: current_term: 1 leader_uuid: "a8d07999846d46a8b293bdb7ba550321" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8d07999846d46a8b293bdb7ba550321" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40613 } } }
I20260928 15:01:36.413714 20111 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8d07999846d46a8b293bdb7ba550321 [sys.catalog]: This master's current role is: LEADER
I20260928 15:01:36.414337 20110 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8d07999846d46a8b293bdb7ba550321 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8d07999846d46a8b293bdb7ba550321" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8d07999846d46a8b293bdb7ba550321" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40613 } } }
I20260928 15:01:36.414672 20110 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8d07999846d46a8b293bdb7ba550321 [sys.catalog]: This master's current role is: LEADER
I20260928 15:01:36.418713 20118 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:01:36.425648 20118 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:01:36.435159 20118 catalog_manager.cc:1347] Generated new cluster ID: f9d1856fcb6743559bc014207470ee9d
I20260928 15:01:36.435305 20118 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:01:36.461024 20118 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:01:36.463089 20118 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:01:36.477339 20118 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8d07999846d46a8b293bdb7ba550321: Generated new TSK 0
I20260928 15:01:36.478361 20118 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:01:36.494794 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 20032
W20260928 15:01:36.496443 20094 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37249
W20260928 15:01:36.497272 31904 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 20032: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:01:36.497483 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 20032
W20260928 15:01:36.515507 20017 connection.cc:535] client connection to 127.0.0.1:40613 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40613): Connection reset by peer (error 104)
2026-09-28T15:01:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2004 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607365362981-31904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:01:36 dist-test-slave-j968 krb5kdc[20131](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:01:36 dist-test-slave-j968 krb5kdc[20131](info): set up 2 sockets
Sep 28 15:01:36 dist-test-slave-j968 krb5kdc[20131](info): commencing operation
krb5kdc: starting...
W20260928 15:01:37.617172 31904 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Sep 28 15:01:37 dist-test-slave-j968 krb5kdc[20131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607697, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:01:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:01:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607365362981-31904-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607365362981-31904-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:01:37.883641 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607365362981-31904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607365362981-31904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607365362981-31904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39949
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607365362981-31904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39949
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:01:38.226502 20144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:38.226874 20144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:38.227065 20144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:38.234335 20144 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:01:38.234510 20144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:38.234620 20144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:38.234730 20144 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:01:38.234815 20144 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:01:38.244889 20144 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607365362981-31904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607365362981-31904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39949
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607365362981-31904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607365362981-31904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:01:38.246275 20144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:38.247622 20144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:38.260118 20149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:38.260098 20150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:38.260205 20152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:38.260957 20144 server_base.cc:1034] running on GCE node
I20260928 15:01:38.261731 20144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:38.263181 20144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:38.264411 20144 hybrid_clock.cc:648] HybridClock initialized: now 1790607698264364 us; error 52 us; skew 500 ppm
I20260928 15:01:38.264757 20144 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:01:38 dist-test-slave-j968 krb5kdc[20131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607698, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:01:38.276342 20144 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:01:38.278030 20144 webserver.cc:460] Webserver started at http://127.0.0.1:39915/ using document root <none> and password file <none>
I20260928 15:01:38.278757 20144 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:38.278914 20144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:38.279235 20144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:38.282358 20144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607365362981-31904-0/minicluster-data/master-0/data/instance:
uuid: "2c92eb411e2b4a199dda322477eaa1c0"
format_stamp: "Formatted at 2026-09-28 15:01:38 on dist-test-slave-j968"
server_key: "38b5afbcaa21e0af2288c09e1205fec3"
server_key_iv: "1db41eeaf3997657f0bd895689ce78d7"
server_key_version: "encryptionkey@0"
I20260928 15:01:38.283128 20144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607365362981-31904-0/minicluster-data/master-0/wal/instance:
uuid: "2c92eb411e2b4a199dda322477eaa1c0"
format_stamp: "Formatted at 2026-09-28 15:01:38 on dist-test-slave-j968"
server_key: "38b5afbcaa21e0af2288c09e1205fec3"
server_key_iv: "1db41eeaf3997657f0bd895689ce78d7"
server_key_version: "encryptionkey@0"
I20260928 15:01:38.288482 20144 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260928 15:01:38.292014 20159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:38.293409 20144 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:01:38.293617 20144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607365362981-31904-0/minicluster-data/master-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607365362981-31904-0/minicluster-data/master-0/wal
uuid: "2c92eb411e2b4a199dda322477eaa1c0"
format_stamp: "Formatted at 2026-09-28 15:01:38 on dist-test-slave-j968"
server_key: "38b5afbcaa21e0af2288c09e1205fec3"
server_key_iv: "1db41eeaf3997657f0bd895689ce78d7"
server_key_version: "encryptionkey@0"
I20260928 15:01:38.293849 20144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607365362981-31904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607365362981-31904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607365362981-31904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:38.299813 20144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:38.304436 20144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:38.304746 20144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:38.324404 20144 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39949
I20260928 15:01:38.324426 20210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39949 every 8 connection(s)
I20260928 15:01:38.325881 20144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
I20260928 15:01:38.329849 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 20144
I20260928 15:01:38.330060 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607365362981-31904-0/minicluster-data/master-0/wal/instance
I20260928 15:01:38.330379 20211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:38.330680 31904 external_mini_cluster.cc:1428] Setting key 129f8596800bca8508a2eab4382fd4e9
I20260928 15:01:38.339185 20214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:38.332644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58295 (local address 127.0.0.1:39949)
0928 15:01:38.333440 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:38.333452 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:38.333494 (+ 42us) server_negotiation.cc:407] Connection header received
0928 15:01:38.335480 (+ 1986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:38.335502 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:38.335857 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:38.336345 (+ 488us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:38.337482 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:38.337488 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:38.337502 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:01:38.337515 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:38.337576 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:01:38.337595 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:38.337850 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:38.338003 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 15:01:38.338187 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":351,"threads_started":1}
I20260928 15:01:38.342999 31904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:01:38.344978 20211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c92eb411e2b4a199dda322477eaa1c0: Bootstrap starting.
I20260928 15:01:38.348796 20211 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c92eb411e2b4a199dda322477eaa1c0: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:38.350041 20214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:38.347611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36907 (local address 127.0.0.1:39949)
0928 15:01:38.347817 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:38.347823 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:38.348081 (+ 258us) server_negotiation.cc:407] Connection header received
0928 15:01:38.348223 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:38.348228 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:38.348294 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:38.348422 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:38.349339 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:38.349344 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:38.349366 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:01:38.349386 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:38.349437 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:01:38.349459 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:38.349576 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:38.349652 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 15:01:38.349710 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 15:01:38.350304 20211 log.cc:824] T 00000000000000000000000000000000 P 2c92eb411e2b4a199dda322477eaa1c0: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:38.353364 20211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c92eb411e2b4a199dda322477eaa1c0: No bootstrap required, opened a new log
I20260928 15:01:38.361896 20211 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c92eb411e2b4a199dda322477eaa1c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c92eb411e2b4a199dda322477eaa1c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39949 } }
I20260928 15:01:38.362327 20211 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c92eb411e2b4a199dda322477eaa1c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:38.362450 20211 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c92eb411e2b4a199dda322477eaa1c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c92eb411e2b4a199dda322477eaa1c0, State: Initialized, Role: FOLLOWER
I20260928 15:01:38.363168 20211 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c92eb411e2b4a199dda322477eaa1c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c92eb411e2b4a199dda322477eaa1c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39949 } }
I20260928 15:01:38.363502 20211 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c92eb411e2b4a199dda322477eaa1c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:01:38.363662 20211 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c92eb411e2b4a199dda322477eaa1c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:01:38.363905 20211 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c92eb411e2b4a199dda322477eaa1c0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:38.366400 20211 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c92eb411e2b4a199dda322477eaa1c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c92eb411e2b4a199dda322477eaa1c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39949 } }
I20260928 15:01:38.366966 20211 leader_election.cc:302] T 00000000000000000000000000000000 P 2c92eb411e2b4a199dda322477eaa1c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c92eb411e2b4a199dda322477eaa1c0; no voters:
I20260928 15:01:38.367604 20211 leader_election.cc:288] T 00000000000000000000000000000000 P 2c92eb411e2b4a199dda322477eaa1c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:01:38.367782 20221 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c92eb411e2b4a199dda322477eaa1c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:01:38.368537 20221 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c92eb411e2b4a199dda322477eaa1c0 [term 1 LEADER]: Becoming Leader. State: Replica: 2c92eb411e2b4a199dda322477eaa1c0, State: Running, Role: LEADER
I20260928 15:01:38.369127 20221 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c92eb411e2b4a199dda322477eaa1c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c92eb411e2b4a199dda322477eaa1c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39949 } }
I20260928 15:01:38.370056 20211 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c92eb411e2b4a199dda322477eaa1c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:01:38.372779 20222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c92eb411e2b4a199dda322477eaa1c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c92eb411e2b4a199dda322477eaa1c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c92eb411e2b4a199dda322477eaa1c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39949 } } }
I20260928 15:01:38.373328 20222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c92eb411e2b4a199dda322477eaa1c0 [sys.catalog]: This master's current role is: LEADER
I20260928 15:01:38.372767 20223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c92eb411e2b4a199dda322477eaa1c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c92eb411e2b4a199dda322477eaa1c0. Latest consensus state: current_term: 1 leader_uuid: "2c92eb411e2b4a199dda322477eaa1c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c92eb411e2b4a199dda322477eaa1c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39949 } } }
I20260928 15:01:38.374006 20223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c92eb411e2b4a199dda322477eaa1c0 [sys.catalog]: This master's current role is: LEADER
I20260928 15:01:38.374773 20228 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:01:38.380743 20228 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:01:38.390686 20228 catalog_manager.cc:1347] Generated new cluster ID: 9762c591cf00443db19fdfff2e28089e
I20260928 15:01:38.390820 20228 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:01:38.422521 20228 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:01:38.423041 20228 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:01:38.432399 20228 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c92eb411e2b4a199dda322477eaa1c0: Generated new TSK 0
I20260928 15:01:38.433650 20228 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:01:38.440714 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 20144
W20260928 15:01:38.441949 20206 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58295
W20260928 15:01:38.442831 31904 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 20144: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:01:38.443001 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 20144
2026-09-28T15:01:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1944 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607365362981-31904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:01:38 dist-test-slave-j968 krb5kdc[20242](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:01:38 dist-test-slave-j968 krb5kdc[20242](info): set up 2 sockets
Sep 28 15:01:38 dist-test-slave-j968 krb5kdc[20242](info): commencing operation
krb5kdc: starting...
W20260928 15:01:39.549142 31904 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.005s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Sep 28 15:01:39 dist-test-slave-j968 krb5kdc[20242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:01:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:01:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607365362981-31904-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607365362981-31904-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:01:39.823371 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607365362981-31904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607365362981-31904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607365362981-31904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39433
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607365362981-31904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39433
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:01:40.171526 20255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:40.171837 20255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:40.172021 20255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:40.179395 20255 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:01:40.179513 20255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:40.179560 20255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:40.179621 20255 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:01:40.179666 20255 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:01:40.190312 20255 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607365362981-31904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607365362981-31904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39433
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607365362981-31904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607365362981-31904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:01:40.191984 20255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:40.193312 20255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:40.205790 20261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:40.205703 20260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:40.205739 20263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:40.206841 20255 server_base.cc:1034] running on GCE node
I20260928 15:01:40.207656 20255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:40.209283 20255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:40.210538 20255 hybrid_clock.cc:648] HybridClock initialized: now 1790607700210476 us; error 49 us; skew 500 ppm
I20260928 15:01:40.210912 20255 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:01:40 dist-test-slave-j968 krb5kdc[20242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607700, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:01:40.214310 20255 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:01:40.216116 20255 webserver.cc:460] Webserver started at http://127.0.0.1:40367/ using document root <none> and password file <none>
I20260928 15:01:40.216825 20255 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:40.216980 20255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:40.217307 20255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:40.220546 20255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607365362981-31904-0/minicluster-data/master-0/data/instance:
uuid: "c4e665f1d5d24cc5aa234389caa1b0c6"
format_stamp: "Formatted at 2026-09-28 15:01:40 on dist-test-slave-j968"
server_key: "4a69ad6e93eaa88b993ff11cdcfd5d71"
server_key_iv: "c857b73209beee9eabf02a358f1c14b3"
server_key_version: "encryptionkey@0"
I20260928 15:01:40.221397 20255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607365362981-31904-0/minicluster-data/master-0/wal/instance:
uuid: "c4e665f1d5d24cc5aa234389caa1b0c6"
format_stamp: "Formatted at 2026-09-28 15:01:40 on dist-test-slave-j968"
server_key: "4a69ad6e93eaa88b993ff11cdcfd5d71"
server_key_iv: "c857b73209beee9eabf02a358f1c14b3"
server_key_version: "encryptionkey@0"
I20260928 15:01:40.226984 20255 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260928 15:01:40.230646 20270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:40.232164 20255 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:01:40.232369 20255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607365362981-31904-0/minicluster-data/master-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607365362981-31904-0/minicluster-data/master-0/wal
uuid: "c4e665f1d5d24cc5aa234389caa1b0c6"
format_stamp: "Formatted at 2026-09-28 15:01:40 on dist-test-slave-j968"
server_key: "4a69ad6e93eaa88b993ff11cdcfd5d71"
server_key_iv: "c857b73209beee9eabf02a358f1c14b3"
server_key_version: "encryptionkey@0"
I20260928 15:01:40.232579 20255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607365362981-31904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607365362981-31904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607365362981-31904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:40.253376 20255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:40.258118 20255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:40.258431 20255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:40.277698 20255 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39433
I20260928 15:01:40.277745 20321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39433 every 8 connection(s)
I20260928 15:01:40.279186 20255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
I20260928 15:01:40.280893 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 20255
I20260928 15:01:40.281086 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607365362981-31904-0/minicluster-data/master-0/wal/instance
I20260928 15:01:40.281636 31904 external_mini_cluster.cc:1428] Setting key 60438744b9c082a1b315db36f6d7775b
I20260928 15:01:40.284019 20322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 15:01:40 dist-test-slave-j968 krb5kdc[20242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 15:01:40.297464 20322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4e665f1d5d24cc5aa234389caa1b0c6: Bootstrap starting.
I20260928 15:01:40.301744 20322 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4e665f1d5d24cc5aa234389caa1b0c6: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:40.302989 20322 log.cc:824] T 00000000000000000000000000000000 P c4e665f1d5d24cc5aa234389caa1b0c6: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:40.306109 20322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4e665f1d5d24cc5aa234389caa1b0c6: No bootstrap required, opened a new log
I20260928 15:01:40.307062 20325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:40.283479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55289 (local address 127.0.0.1:39433)
0928 15:01:40.284379 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:40.284391 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:40.284432 (+ 41us) server_negotiation.cc:407] Connection header received
0928 15:01:40.286058 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:40.286079 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:40.286471 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:40.286816 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:40.288779 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:40.290345 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:40.292148 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:40.292894 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:40.293046 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:40.298090 (+ 5044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:40.298101 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:40.298120 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:40.298164 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:40.302320 (+ 4156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:40.303276 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:40.303284 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:40.303291 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:40.303483 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:40.304012 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:40.304016 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:40.304019 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:40.304624 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:40.304957 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:40.305473 (+ 516us) server_negotiation.cc:299] Negotiation successful
0928 15:01:40.305948 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":691,"thread_start_us":471,"threads_started":1}
I20260928 15:01:40.311146 31904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:01:40.314352 20322 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4e665f1d5d24cc5aa234389caa1b0c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4e665f1d5d24cc5aa234389caa1b0c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39433 } }
I20260928 15:01:40.314741 20322 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4e665f1d5d24cc5aa234389caa1b0c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:40.314863 20322 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4e665f1d5d24cc5aa234389caa1b0c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4e665f1d5d24cc5aa234389caa1b0c6, State: Initialized, Role: FOLLOWER
I20260928 15:01:40.315443 20322 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4e665f1d5d24cc5aa234389caa1b0c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4e665f1d5d24cc5aa234389caa1b0c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39433 } }
I20260928 15:01:40.315687 20322 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4e665f1d5d24cc5aa234389caa1b0c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:01:40.315829 20322 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4e665f1d5d24cc5aa234389caa1b0c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:01:40.316087 20322 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4e665f1d5d24cc5aa234389caa1b0c6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:40.318663 20322 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4e665f1d5d24cc5aa234389caa1b0c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4e665f1d5d24cc5aa234389caa1b0c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39433 } }
I20260928 15:01:40.319344 20322 leader_election.cc:302] T 00000000000000000000000000000000 P c4e665f1d5d24cc5aa234389caa1b0c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4e665f1d5d24cc5aa234389caa1b0c6; no voters:
I20260928 15:01:40.320094 20322 leader_election.cc:288] T 00000000000000000000000000000000 P c4e665f1d5d24cc5aa234389caa1b0c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:01:40.320214 20332 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4e665f1d5d24cc5aa234389caa1b0c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:01:40.322801 20322 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4e665f1d5d24cc5aa234389caa1b0c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:01:40.324156 20332 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4e665f1d5d24cc5aa234389caa1b0c6 [term 1 LEADER]: Becoming Leader. State: Replica: c4e665f1d5d24cc5aa234389caa1b0c6, State: Running, Role: LEADER
I20260928 15:01:40.324970 20332 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4e665f1d5d24cc5aa234389caa1b0c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4e665f1d5d24cc5aa234389caa1b0c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39433 } }
I20260928 15:01:40.333875 20334 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4e665f1d5d24cc5aa234389caa1b0c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4e665f1d5d24cc5aa234389caa1b0c6. Latest consensus state: current_term: 1 leader_uuid: "c4e665f1d5d24cc5aa234389caa1b0c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4e665f1d5d24cc5aa234389caa1b0c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39433 } } }
I20260928 15:01:40.334340 20334 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4e665f1d5d24cc5aa234389caa1b0c6 [sys.catalog]: This master's current role is: LEADER
I20260928 15:01:40.334888 20333 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4e665f1d5d24cc5aa234389caa1b0c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4e665f1d5d24cc5aa234389caa1b0c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4e665f1d5d24cc5aa234389caa1b0c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39433 } } }
I20260928 15:01:40.335172 20333 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4e665f1d5d24cc5aa234389caa1b0c6 [sys.catalog]: This master's current role is: LEADER
I20260928 15:01:40.337239 20325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:40.316067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37195 (local address 127.0.0.1:39433)
0928 15:01:40.316271 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:40.316277 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:40.316926 (+ 649us) server_negotiation.cc:407] Connection header received
0928 15:01:40.317049 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:40.317059 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:40.317141 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:40.317255 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:40.319000 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:40.320219 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:40.322696 (+ 2477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:40.323492 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:40.323645 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:40.327141 (+ 3496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:40.327151 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:40.327158 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:40.327215 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:40.332313 (+ 5098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:40.333336 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:40.333343 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:40.333347 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:40.333559 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:40.334051 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:40.334056 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:40.334066 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:40.334480 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:40.334801 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:40.336310 (+ 1509us) server_negotiation.cc:299] Negotiation successful
0928 15:01:40.336628 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":67}
I20260928 15:01:40.340986 20342 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:01:40.346804 20342 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:01:40.355825 20342 catalog_manager.cc:1347] Generated new cluster ID: d000b9e612b148d8abe25f35cd949c59
I20260928 15:01:40.355978 20342 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:01:40.390956 20342 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:01:40.392678 20342 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:01:40.400820 20342 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4e665f1d5d24cc5aa234389caa1b0c6: Generated new TSK 0
I20260928 15:01:40.402046 20342 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:01:40.421384 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 20255
I20260928 15:01:40.422700 20317 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:01:40.522781 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 20255
2026-09-28T15:01:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2077 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607365362981-31904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:01:40 dist-test-slave-j968 krb5kdc[20354](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:01:40 dist-test-slave-j968 krb5kdc[20354](info): set up 2 sockets
Sep 28 15:01:40 dist-test-slave-j968 krb5kdc[20354](info): commencing operation
krb5kdc: starting...
W20260928 15:01:41.633281 31904 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Sep 28 15:01:41 dist-test-slave-j968 krb5kdc[20354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:01:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:01:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607365362981-31904-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607365362981-31904-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:01:41.912061 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607365362981-31904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607365362981-31904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607365362981-31904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37209
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607365362981-31904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37209
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:01:42.257084 20367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:42.257480 20367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:42.257656 20367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:42.265118 20367 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:01:42.265295 20367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:42.265407 20367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:42.265518 20367 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:01:42.265602 20367 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:01:42.275635 20367 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607365362981-31904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607365362981-31904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607365362981-31904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607365362981-31904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:01:42.277079 20367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:42.278496 20367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:42.290767 20375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:42.290872 20373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:42.292048 20372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:42.296129 20367 server_base.cc:1034] running on GCE node
I20260928 15:01:42.296916 20367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:42.298355 20367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:42.299599 20367 hybrid_clock.cc:648] HybridClock initialized: now 1790607702299554 us; error 38 us; skew 500 ppm
I20260928 15:01:42.300019 20367 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:01:42 dist-test-slave-j968 krb5kdc[20354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607702, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:01:42.303228 20367 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:01:42.304919 20367 webserver.cc:460] Webserver started at http://127.0.0.1:38383/ using document root <none> and password file <none>
I20260928 15:01:42.305663 20367 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:42.305811 20367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:42.306133 20367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:42.309271 20367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607365362981-31904-0/minicluster-data/master-0/data/instance:
uuid: "957efb92421745d3a47ad7d52a12fa45"
format_stamp: "Formatted at 2026-09-28 15:01:42 on dist-test-slave-j968"
server_key: "d5b26667262897821fd7eeb7cddd872e"
server_key_iv: "4c817d5535146469acfb5b6a024bbb9f"
server_key_version: "encryptionkey@0"
I20260928 15:01:42.310073 20367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607365362981-31904-0/minicluster-data/master-0/wal/instance:
uuid: "957efb92421745d3a47ad7d52a12fa45"
format_stamp: "Formatted at 2026-09-28 15:01:42 on dist-test-slave-j968"
server_key: "d5b26667262897821fd7eeb7cddd872e"
server_key_iv: "4c817d5535146469acfb5b6a024bbb9f"
server_key_version: "encryptionkey@0"
I20260928 15:01:42.315717 20367 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260928 15:01:42.319448 20382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:42.321292 20367 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:01:42.321543 20367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607365362981-31904-0/minicluster-data/master-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607365362981-31904-0/minicluster-data/master-0/wal
uuid: "957efb92421745d3a47ad7d52a12fa45"
format_stamp: "Formatted at 2026-09-28 15:01:42 on dist-test-slave-j968"
server_key: "d5b26667262897821fd7eeb7cddd872e"
server_key_iv: "4c817d5535146469acfb5b6a024bbb9f"
server_key_version: "encryptionkey@0"
I20260928 15:01:42.321969 20367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607365362981-31904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607365362981-31904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607365362981-31904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:42.348654 20367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:42.353102 20367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:42.353430 20367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:42.372833 20367 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37209
I20260928 15:01:42.372884 20433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37209 every 8 connection(s)
I20260928 15:01:42.374341 20367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
I20260928 15:01:42.378489 20434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:42.378767 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 20367
I20260928 15:01:42.378957 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607365362981-31904-0/minicluster-data/master-0/wal/instance
I20260928 15:01:42.379400 31904 external_mini_cluster.cc:1428] Setting key ff984c4d0c02bda835fdc49de7f7ad04
I20260928 15:01:42.392016 20434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 957efb92421745d3a47ad7d52a12fa45: Bootstrap starting.
I20260928 15:01:42.393498 20437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:42.381293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34057 (local address 127.0.0.1:37209)
0928 15:01:42.382188 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:42.382199 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:42.382242 (+ 43us) server_negotiation.cc:407] Connection header received
0928 15:01:42.384377 (+ 2135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:42.384402 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:42.384953 (+ 551us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:42.385342 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:42.385883 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:42.387137 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:42.389098 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:42.389782 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:42.389916 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:42.391138 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:42.391145 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:42.391159 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:01:42.391701 (+ 542us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:42.391774 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:01:42.391799 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:42.391961 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:42.392199 (+ 238us) server_negotiation.cc:299] Negotiation successful
0928 15:01:42.392532 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":433,"threads_started":1}
I20260928 15:01:42.395613 20434 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 957efb92421745d3a47ad7d52a12fa45: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:42.396983 20434 log.cc:824] T 00000000000000000000000000000000 P 957efb92421745d3a47ad7d52a12fa45: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:42.397588 31904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:01:42.399726 20434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 957efb92421745d3a47ad7d52a12fa45: No bootstrap required, opened a new log
I20260928 15:01:42.408471 20437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:42.401997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36291 (local address 127.0.0.1:37209)
0928 15:01:42.402184 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:42.402194 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:42.402540 (+ 346us) server_negotiation.cc:407] Connection header received
0928 15:01:42.402688 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:42.402693 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:42.402766 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:42.402882 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:42.403347 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:42.404491 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:42.405804 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:42.406503 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:42.406628 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:42.407660 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:42.407667 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:42.407673 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:01:42.407696 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:42.407741 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:01:42.407761 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:42.407857 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:42.408038 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 15:01:42.408131 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 15:01:42.409473 20434 raft_consensus.cc:348] T 00000000000000000000000000000000 P 957efb92421745d3a47ad7d52a12fa45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "957efb92421745d3a47ad7d52a12fa45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37209 } }
I20260928 15:01:42.409968 20434 raft_consensus.cc:374] T 00000000000000000000000000000000 P 957efb92421745d3a47ad7d52a12fa45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:42.410097 20434 raft_consensus.cc:729] T 00000000000000000000000000000000 P 957efb92421745d3a47ad7d52a12fa45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 957efb92421745d3a47ad7d52a12fa45, State: Initialized, Role: FOLLOWER
I20260928 15:01:42.410681 20434 consensus_queue.cc:260] T 00000000000000000000000000000000 P 957efb92421745d3a47ad7d52a12fa45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "957efb92421745d3a47ad7d52a12fa45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37209 } }
I20260928 15:01:42.410939 20434 raft_consensus.cc:388] T 00000000000000000000000000000000 P 957efb92421745d3a47ad7d52a12fa45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:01:42.411085 20434 raft_consensus.cc:482] T 00000000000000000000000000000000 P 957efb92421745d3a47ad7d52a12fa45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:01:42.411273 20434 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 957efb92421745d3a47ad7d52a12fa45 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:42.413537 20434 raft_consensus.cc:504] T 00000000000000000000000000000000 P 957efb92421745d3a47ad7d52a12fa45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "957efb92421745d3a47ad7d52a12fa45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37209 } }
I20260928 15:01:42.414145 20434 leader_election.cc:302] T 00000000000000000000000000000000 P 957efb92421745d3a47ad7d52a12fa45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 957efb92421745d3a47ad7d52a12fa45; no voters:
I20260928 15:01:42.414741 20434 leader_election.cc:288] T 00000000000000000000000000000000 P 957efb92421745d3a47ad7d52a12fa45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:01:42.414925 20444 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 957efb92421745d3a47ad7d52a12fa45 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:01:42.415638 20444 raft_consensus.cc:686] T 00000000000000000000000000000000 P 957efb92421745d3a47ad7d52a12fa45 [term 1 LEADER]: Becoming Leader. State: Replica: 957efb92421745d3a47ad7d52a12fa45, State: Running, Role: LEADER
I20260928 15:01:42.416237 20444 consensus_queue.cc:237] T 00000000000000000000000000000000 P 957efb92421745d3a47ad7d52a12fa45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "957efb92421745d3a47ad7d52a12fa45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37209 } }
I20260928 15:01:42.417127 20434 sys_catalog.cc:564] T 00000000000000000000000000000000 P 957efb92421745d3a47ad7d52a12fa45 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:01:42.419441 20445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 957efb92421745d3a47ad7d52a12fa45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "957efb92421745d3a47ad7d52a12fa45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "957efb92421745d3a47ad7d52a12fa45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37209 } } }
I20260928 15:01:42.419842 20445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 957efb92421745d3a47ad7d52a12fa45 [sys.catalog]: This master's current role is: LEADER
I20260928 15:01:42.420442 20446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 957efb92421745d3a47ad7d52a12fa45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 957efb92421745d3a47ad7d52a12fa45. Latest consensus state: current_term: 1 leader_uuid: "957efb92421745d3a47ad7d52a12fa45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "957efb92421745d3a47ad7d52a12fa45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37209 } } }
I20260928 15:01:42.420820 20446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 957efb92421745d3a47ad7d52a12fa45 [sys.catalog]: This master's current role is: LEADER
I20260928 15:01:42.428972 20450 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:01:42.434098 20450 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:01:42.443302 20450 catalog_manager.cc:1347] Generated new cluster ID: 876c0780e8ca4ae2a8d7477cfa0e4b02
I20260928 15:01:42.443502 20450 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:01:42.478140 20450 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:01:42.479569 20450 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:01:42.500056 20450 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 957efb92421745d3a47ad7d52a12fa45: Generated new TSK 0
I20260928 15:01:42.501052 20450 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:01:42.511976 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 20367
W20260928 15:01:42.513046 20429 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34057
W20260928 15:01:42.513759 31904 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 20367: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:01:42.513916 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 20367
2026-09-28T15:01:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1991 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607365362981-31904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:01:42 dist-test-slave-j968 krb5kdc[20466](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:01:42 dist-test-slave-j968 krb5kdc[20466](info): set up 2 sockets
Sep 28 15:01:42 dist-test-slave-j968 krb5kdc[20466](info): commencing operation
krb5kdc: starting...
W20260928 15:01:43.637399 31904 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Sep 28 15:01:43 dist-test-slave-j968 krb5kdc[20466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:01:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:01:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607365362981-31904-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607365362981-31904-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:01:43.915403 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607365362981-31904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607365362981-31904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607365362981-31904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42375
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607365362981-31904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42375
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:01:44.260538 20479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:44.260836 20479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:44.261011 20479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:44.268280 20479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:01:44.268394 20479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:44.268503 20479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:44.268643 20479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:01:44.268725 20479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:01:44.278637 20479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607365362981-31904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607365362981-31904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42375
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607365362981-31904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607365362981-31904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:01:44.280041 20479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:44.281404 20479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:44.294390 20485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:44.294875 20487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:44.295100 20484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:44.295645 20479 server_base.cc:1034] running on GCE node
I20260928 15:01:44.296747 20479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:44.298784 20479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:44.300041 20479 hybrid_clock.cc:648] HybridClock initialized: now 1790607704299955 us; error 87 us; skew 500 ppm
I20260928 15:01:44.300531 20479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:01:44 dist-test-slave-j968 krb5kdc[20466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607704, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:01:44.310020 20479 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:01:44.311655 20479 webserver.cc:460] Webserver started at http://127.0.0.1:35647/ using document root <none> and password file <none>
I20260928 15:01:44.312443 20479 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:44.312592 20479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:44.312930 20479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:44.316159 20479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607365362981-31904-0/minicluster-data/master-0/data/instance:
uuid: "3eb0e8ba99914861abd21b2573c5a344"
format_stamp: "Formatted at 2026-09-28 15:01:44 on dist-test-slave-j968"
server_key: "bf8d255e237f15ac08d589dc43f79562"
server_key_iv: "e1545292612dd2b5f2022f1f946c9535"
server_key_version: "encryptionkey@0"
I20260928 15:01:44.316941 20479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607365362981-31904-0/minicluster-data/master-0/wal/instance:
uuid: "3eb0e8ba99914861abd21b2573c5a344"
format_stamp: "Formatted at 2026-09-28 15:01:44 on dist-test-slave-j968"
server_key: "bf8d255e237f15ac08d589dc43f79562"
server_key_iv: "e1545292612dd2b5f2022f1f946c9535"
server_key_version: "encryptionkey@0"
I20260928 15:01:44.322352 20479 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260928 15:01:44.326215 20494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:44.328027 20479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:01:44.328217 20479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607365362981-31904-0/minicluster-data/master-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607365362981-31904-0/minicluster-data/master-0/wal
uuid: "3eb0e8ba99914861abd21b2573c5a344"
format_stamp: "Formatted at 2026-09-28 15:01:44 on dist-test-slave-j968"
server_key: "bf8d255e237f15ac08d589dc43f79562"
server_key_iv: "e1545292612dd2b5f2022f1f946c9535"
server_key_version: "encryptionkey@0"
I20260928 15:01:44.328442 20479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607365362981-31904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607365362981-31904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607365362981-31904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:44.334422 20479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:44.339285 20479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:44.339596 20479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:44.358632 20479 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42375
I20260928 15:01:44.358681 20545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42375 every 8 connection(s)
I20260928 15:01:44.360206 20479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
I20260928 15:01:44.361958 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 20479
I20260928 15:01:44.362155 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607365362981-31904-0/minicluster-data/master-0/wal/instance
I20260928 15:01:44.362730 31904 external_mini_cluster.cc:1428] Setting key 95a70f7409553f8622ffa3f669ddbf48
I20260928 15:01:44.365044 20546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:44.371896 20549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:44.364592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55669 (local address 127.0.0.1:42375)
0928 15:01:44.365417 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:44.365435 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:44.365486 (+ 51us) server_negotiation.cc:407] Connection header received
0928 15:01:44.367810 (+ 2324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:44.367838 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:44.368351 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:44.368833 (+ 482us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:44.369859 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:44.369868 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:44.369886 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:01:44.369919 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:44.369998 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:01:44.370027 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:44.370239 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:44.370375 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 15:01:44.370579 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":311,"threads_started":1}
I20260928 15:01:44.375756 31904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:01:44.379290 20546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3eb0e8ba99914861abd21b2573c5a344: Bootstrap starting.
I20260928 15:01:44.383351 20546 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3eb0e8ba99914861abd21b2573c5a344: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:44.384272 20549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:44.381420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43783 (local address 127.0.0.1:42375)
0928 15:01:44.381629 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:44.381636 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:44.382196 (+ 560us) server_negotiation.cc:407] Connection header received
0928 15:01:44.382338 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:44.382344 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:44.382421 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:44.382548 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:44.383560 (+ 1012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:44.383565 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:44.383570 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:01:44.383596 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:44.383636 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:01:44.383652 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:44.383782 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:44.383928 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 15:01:44.383990 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 15:01:44.384874 20546 log.cc:824] T 00000000000000000000000000000000 P 3eb0e8ba99914861abd21b2573c5a344: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:44.387759 20546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3eb0e8ba99914861abd21b2573c5a344: No bootstrap required, opened a new log
I20260928 15:01:44.395126 20546 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3eb0e8ba99914861abd21b2573c5a344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eb0e8ba99914861abd21b2573c5a344" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42375 } }
I20260928 15:01:44.395545 20546 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3eb0e8ba99914861abd21b2573c5a344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:44.395655 20546 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3eb0e8ba99914861abd21b2573c5a344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb0e8ba99914861abd21b2573c5a344, State: Initialized, Role: FOLLOWER
I20260928 15:01:44.396296 20546 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3eb0e8ba99914861abd21b2573c5a344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eb0e8ba99914861abd21b2573c5a344" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42375 } }
I20260928 15:01:44.396553 20546 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3eb0e8ba99914861abd21b2573c5a344 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:01:44.396729 20546 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3eb0e8ba99914861abd21b2573c5a344 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:01:44.396919 20546 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3eb0e8ba99914861abd21b2573c5a344 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:44.399150 20546 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3eb0e8ba99914861abd21b2573c5a344 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eb0e8ba99914861abd21b2573c5a344" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42375 } }
I20260928 15:01:44.399726 20546 leader_election.cc:302] T 00000000000000000000000000000000 P 3eb0e8ba99914861abd21b2573c5a344 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3eb0e8ba99914861abd21b2573c5a344; no voters:
I20260928 15:01:44.400372 20546 leader_election.cc:288] T 00000000000000000000000000000000 P 3eb0e8ba99914861abd21b2573c5a344 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:01:44.400537 20556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3eb0e8ba99914861abd21b2573c5a344 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:01:44.401273 20556 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3eb0e8ba99914861abd21b2573c5a344 [term 1 LEADER]: Becoming Leader. State: Replica: 3eb0e8ba99914861abd21b2573c5a344, State: Running, Role: LEADER
I20260928 15:01:44.401861 20556 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3eb0e8ba99914861abd21b2573c5a344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eb0e8ba99914861abd21b2573c5a344" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42375 } }
I20260928 15:01:44.402613 20546 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3eb0e8ba99914861abd21b2573c5a344 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:01:44.407996 20558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3eb0e8ba99914861abd21b2573c5a344 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3eb0e8ba99914861abd21b2573c5a344. Latest consensus state: current_term: 1 leader_uuid: "3eb0e8ba99914861abd21b2573c5a344" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eb0e8ba99914861abd21b2573c5a344" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42375 } } }
I20260928 15:01:44.407994 20557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3eb0e8ba99914861abd21b2573c5a344 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3eb0e8ba99914861abd21b2573c5a344" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eb0e8ba99914861abd21b2573c5a344" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42375 } } }
I20260928 15:01:44.408592 20558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3eb0e8ba99914861abd21b2573c5a344 [sys.catalog]: This master's current role is: LEADER
I20260928 15:01:44.408608 20557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3eb0e8ba99914861abd21b2573c5a344 [sys.catalog]: This master's current role is: LEADER
I20260928 15:01:44.410472 20565 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:01:44.416767 20565 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:01:44.427327 20565 catalog_manager.cc:1347] Generated new cluster ID: 4b00181de132412cbe06fca983ea006d
I20260928 15:01:44.427457 20565 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:01:44.457321 20565 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:01:44.457803 20565 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:01:44.469543 20565 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3eb0e8ba99914861abd21b2573c5a344: Generated new TSK 0
I20260928 15:01:44.470438 20565 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:01:44.481648 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 20479
W20260928 15:01:44.483707 20541 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55669
W20260928 15:01:44.484547 31904 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 20479: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:01:44.484797 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 20479
2026-09-28T15:01:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1974 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607365362981-31904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:01:44 dist-test-slave-j968 krb5kdc[20577](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:01:44 dist-test-slave-j968 krb5kdc[20577](info): set up 2 sockets
Sep 28 15:01:44 dist-test-slave-j968 krb5kdc[20577](info): commencing operation
krb5kdc: starting...
W20260928 15:01:45.601279 31904 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Sep 28 15:01:45 dist-test-slave-j968 krb5kdc[20577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607705, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:01:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:01:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607365362981-31904-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607365362981-31904-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:01:45.882735 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607365362981-31904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607365362981-31904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607365362981-31904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35905
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607365362981-31904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35905
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:01:46.247460 20590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:46.247828 20590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:46.248001 20590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:46.255324 20590 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:01:46.255503 20590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:46.255591 20590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:46.255682 20590 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:01:46.255749 20590 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:01:46.265846 20590 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607365362981-31904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607365362981-31904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35905
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607365362981-31904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607365362981-31904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:01:46.267280 20590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:46.268687 20590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:46.280973 20596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:46.281013 20598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:46.280933 20595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:46.281730 20590 server_base.cc:1034] running on GCE node
I20260928 15:01:46.282564 20590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:46.284114 20590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:46.285354 20590 hybrid_clock.cc:648] HybridClock initialized: now 1790607706285309 us; error 52 us; skew 500 ppm
I20260928 15:01:46.285713 20590 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:01:46 dist-test-slave-j968 krb5kdc[20577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607706, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:01:46.288934 20590 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:01:46.290477 20590 webserver.cc:460] Webserver started at http://127.0.0.1:36255/ using document root <none> and password file <none>
I20260928 15:01:46.291179 20590 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:46.291332 20590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:46.291636 20590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:46.294709 20590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607365362981-31904-0/minicluster-data/master-0/data/instance:
uuid: "4e60906aee5b407692c4a7cde6f732b7"
format_stamp: "Formatted at 2026-09-28 15:01:46 on dist-test-slave-j968"
server_key: "7d2facfd43b1cfb066eb2403c0a8be77"
server_key_iv: "592dc6d923551dde32718d2c2353e7da"
server_key_version: "encryptionkey@0"
I20260928 15:01:46.295495 20590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607365362981-31904-0/minicluster-data/master-0/wal/instance:
uuid: "4e60906aee5b407692c4a7cde6f732b7"
format_stamp: "Formatted at 2026-09-28 15:01:46 on dist-test-slave-j968"
server_key: "7d2facfd43b1cfb066eb2403c0a8be77"
server_key_iv: "592dc6d923551dde32718d2c2353e7da"
server_key_version: "encryptionkey@0"
I20260928 15:01:46.300923 20590 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 15:01:46.304812 20605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:46.306293 20590 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:01:46.306517 20590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607365362981-31904-0/minicluster-data/master-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607365362981-31904-0/minicluster-data/master-0/wal
uuid: "4e60906aee5b407692c4a7cde6f732b7"
format_stamp: "Formatted at 2026-09-28 15:01:46 on dist-test-slave-j968"
server_key: "7d2facfd43b1cfb066eb2403c0a8be77"
server_key_iv: "592dc6d923551dde32718d2c2353e7da"
server_key_version: "encryptionkey@0"
I20260928 15:01:46.306756 20590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607365362981-31904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607365362981-31904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607365362981-31904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:46.338419 20590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:46.343293 20590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:46.343611 20590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:46.363035 20590 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35905
I20260928 15:01:46.363063 20656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35905 every 8 connection(s)
I20260928 15:01:46.364490 20590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
I20260928 15:01:46.368602 20657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:46.369709 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 20590
I20260928 15:01:46.369904 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607365362981-31904-0/minicluster-data/master-0/wal/instance
I20260928 15:01:46.370469 31904 external_mini_cluster.cc:1428] Setting key 570586d7699be59a4cc10e29ea82945d
I20260928 15:01:46.383289 20657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e60906aee5b407692c4a7cde6f732b7: Bootstrap starting.
Sep 28 15:01:46 dist-test-slave-j968 krb5kdc[20577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607705, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 15:01:46.387058 20657 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e60906aee5b407692c4a7cde6f732b7: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:46.388303 20657 log.cc:824] T 00000000000000000000000000000000 P 4e60906aee5b407692c4a7cde6f732b7: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:46.391443 20657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e60906aee5b407692c4a7cde6f732b7: No bootstrap required, opened a new log
I20260928 15:01:46.394438 20660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:46.372442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:47463 (local address 127.0.0.1:35905)
0928 15:01:46.373241 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:46.373253 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:46.373303 (+ 50us) server_negotiation.cc:407] Connection header received
0928 15:01:46.374989 (+ 1686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:46.375016 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:46.375451 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:46.375793 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:46.377691 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:46.379037 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:46.380894 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:46.381566 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:46.381718 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:46.386621 (+ 4903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:46.386629 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:46.386645 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:46.386688 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:46.390201 (+ 3513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:46.391029 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:46.391036 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:46.391042 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:46.391206 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:46.391704 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:46.391708 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:46.391710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:46.392271 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:46.392632 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:46.393075 (+ 443us) server_negotiation.cc:299] Negotiation successful
0928 15:01:46.393413 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":369,"threads_started":1}
I20260928 15:01:46.398479 31904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:01:46.399997 20657 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e60906aee5b407692c4a7cde6f732b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e60906aee5b407692c4a7cde6f732b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35905 } }
I20260928 15:01:46.400384 20657 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e60906aee5b407692c4a7cde6f732b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:46.400494 20657 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e60906aee5b407692c4a7cde6f732b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e60906aee5b407692c4a7cde6f732b7, State: Initialized, Role: FOLLOWER
I20260928 15:01:46.401146 20657 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e60906aee5b407692c4a7cde6f732b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e60906aee5b407692c4a7cde6f732b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35905 } }
I20260928 15:01:46.401396 20657 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e60906aee5b407692c4a7cde6f732b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:01:46.401551 20657 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e60906aee5b407692c4a7cde6f732b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:01:46.401767 20657 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e60906aee5b407692c4a7cde6f732b7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:46.404510 20657 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e60906aee5b407692c4a7cde6f732b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e60906aee5b407692c4a7cde6f732b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35905 } }
I20260928 15:01:46.405081 20657 leader_election.cc:302] T 00000000000000000000000000000000 P 4e60906aee5b407692c4a7cde6f732b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e60906aee5b407692c4a7cde6f732b7; no voters:
I20260928 15:01:46.405682 20657 leader_election.cc:288] T 00000000000000000000000000000000 P 4e60906aee5b407692c4a7cde6f732b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:01:46.405922 20667 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e60906aee5b407692c4a7cde6f732b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:01:46.407419 20667 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e60906aee5b407692c4a7cde6f732b7 [term 1 LEADER]: Becoming Leader. State: Replica: 4e60906aee5b407692c4a7cde6f732b7, State: Running, Role: LEADER
I20260928 15:01:46.408051 20667 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e60906aee5b407692c4a7cde6f732b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e60906aee5b407692c4a7cde6f732b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35905 } }
I20260928 15:01:46.408456 20657 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e60906aee5b407692c4a7cde6f732b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:01:46.417629 20668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e60906aee5b407692c4a7cde6f732b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e60906aee5b407692c4a7cde6f732b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e60906aee5b407692c4a7cde6f732b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35905 } } }
I20260928 15:01:46.418107 20668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e60906aee5b407692c4a7cde6f732b7 [sys.catalog]: This master's current role is: LEADER
I20260928 15:01:46.418612 20669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e60906aee5b407692c4a7cde6f732b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e60906aee5b407692c4a7cde6f732b7. Latest consensus state: current_term: 1 leader_uuid: "4e60906aee5b407692c4a7cde6f732b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e60906aee5b407692c4a7cde6f732b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35905 } } }
I20260928 15:01:46.418906 20669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e60906aee5b407692c4a7cde6f732b7 [sys.catalog]: This master's current role is: LEADER
I20260928 15:01:46.423213 20660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:46.403259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:44361 (local address 127.0.0.1:35905)
0928 15:01:46.403454 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:46.403462 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:46.403933 (+ 471us) server_negotiation.cc:407] Connection header received
0928 15:01:46.404110 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:46.404116 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:46.404207 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:46.404332 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:46.406281 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:46.407572 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:46.409378 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:46.410307 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:46.410539 (+ 232us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:46.414731 (+ 4192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:46.414742 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:46.414750 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:46.414804 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:46.419607 (+ 4803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:46.420826 (+ 1219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:46.420832 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:46.420835 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:46.420982 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:46.421609 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:46.421614 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:46.421618 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:46.421976 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:46.422294 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:46.422665 (+ 371us) server_negotiation.cc:299] Negotiation successful
0928 15:01:46.422850 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 15:01:46.429450 20673 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:01:46.434052 20673 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:01:46.442278 20673 catalog_manager.cc:1347] Generated new cluster ID: 111fd8620b1f414b9416fd6ad10bd067
I20260928 15:01:46.442488 20673 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:01:46.472370 20673 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:01:46.473798 20673 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:01:46.488487 20673 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e60906aee5b407692c4a7cde6f732b7: Generated new TSK 0
I20260928 15:01:46.489488 20673 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:01:46.507913 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 20590
I20260928 15:01:46.509274 20652 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:01:46.604368 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 20590
W20260928 15:01:46.623131 20575 connection.cc:535] client connection to 127.0.0.1:35905 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35905): Connection reset by peer (error 104)
2026-09-28T15:01:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2118 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607365362981-31904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:01:46 dist-test-slave-j968 krb5kdc[20689](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:01:46 dist-test-slave-j968 krb5kdc[20689](info): set up 2 sockets
Sep 28 15:01:46 dist-test-slave-j968 krb5kdc[20689](info): commencing operation
krb5kdc: starting...
W20260928 15:01:47.721196 31904 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Sep 28 15:01:47 dist-test-slave-j968 krb5kdc[20689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:01:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:01:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607365362981-31904-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607365362981-31904-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:01:47.999507 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607365362981-31904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607365362981-31904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607365362981-31904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35419
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607365362981-31904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:01:48.344794 20702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:48.345100 20702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:48.345271 20702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:48.352696 20702 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:01:48.352814 20702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:48.352860 20702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:48.352996 20702 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:01:48.353060 20702 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:01:48.363173 20702 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607365362981-31904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607365362981-31904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607365362981-31904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607365362981-31904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:01:48.364574 20702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:48.365934 20702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:48.379081 20707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:48.379056 20708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:48.378996 20710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:48.380229 20702 server_base.cc:1034] running on GCE node
I20260928 15:01:48.381101 20702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:48.382669 20702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:48.383986 20702 hybrid_clock.cc:648] HybridClock initialized: now 1790607708383926 us; error 43 us; skew 500 ppm
I20260928 15:01:48.384372 20702 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:01:48 dist-test-slave-j968 krb5kdc[20689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607708, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:01:48.387638 20702 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:01:48.389335 20702 webserver.cc:460] Webserver started at http://127.0.0.1:38241/ using document root <none> and password file <none>
I20260928 15:01:48.390064 20702 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:48.390223 20702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:48.390540 20702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:48.393667 20702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607365362981-31904-0/minicluster-data/master-0/data/instance:
uuid: "f27b9fed030148f1a1e894df564d965d"
format_stamp: "Formatted at 2026-09-28 15:01:48 on dist-test-slave-j968"
server_key: "6cc111991aa17ac4a5294171ad167fdc"
server_key_iv: "00d56bc25004406fd40f638aacca8606"
server_key_version: "encryptionkey@0"
I20260928 15:01:48.394479 20702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607365362981-31904-0/minicluster-data/master-0/wal/instance:
uuid: "f27b9fed030148f1a1e894df564d965d"
format_stamp: "Formatted at 2026-09-28 15:01:48 on dist-test-slave-j968"
server_key: "6cc111991aa17ac4a5294171ad167fdc"
server_key_iv: "00d56bc25004406fd40f638aacca8606"
server_key_version: "encryptionkey@0"
I20260928 15:01:48.399776 20702 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 15:01:48.403415 20717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:48.404811 20702 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 15:01:48.405017 20702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607365362981-31904-0/minicluster-data/master-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607365362981-31904-0/minicluster-data/master-0/wal
uuid: "f27b9fed030148f1a1e894df564d965d"
format_stamp: "Formatted at 2026-09-28 15:01:48 on dist-test-slave-j968"
server_key: "6cc111991aa17ac4a5294171ad167fdc"
server_key_iv: "00d56bc25004406fd40f638aacca8606"
server_key_version: "encryptionkey@0"
I20260928 15:01:48.405224 20702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607365362981-31904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607365362981-31904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607365362981-31904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:48.444778 20702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:48.449572 20702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:48.449906 20702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:48.469425 20702 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35419
I20260928 15:01:48.469453 20768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35419 every 8 connection(s)
I20260928 15:01:48.470901 20702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
I20260928 15:01:48.474900 20769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:48.477111 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 20702
I20260928 15:01:48.477305 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607365362981-31904-0/minicluster-data/master-0/wal/instance
I20260928 15:01:48.477909 31904 external_mini_cluster.cc:1428] Setting key 46eb3bb3308b50ee8f036b5b873c55f6
I20260928 15:01:48.486748 20769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f27b9fed030148f1a1e894df564d965d: Bootstrap starting.
I20260928 15:01:48.490068 20769 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f27b9fed030148f1a1e894df564d965d: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:48.491178 20772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:48.479825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:46001 (local address 127.0.0.1:35419)
0928 15:01:48.480631 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:48.480647 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:48.480696 (+ 49us) server_negotiation.cc:407] Connection header received
0928 15:01:48.482419 (+ 1723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:48.482443 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:48.482847 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:48.483254 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:48.483847 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:48.485297 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:48.486653 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:48.487415 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:48.487616 (+ 201us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:48.488939 (+ 1323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:48.488947 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:48.488961 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:01:48.489495 (+ 534us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:48.489569 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:01:48.489595 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:48.489764 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:48.490046 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 15:01:48.490227 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":317,"threads_started":1}
I20260928 15:01:48.491156 20769 log.cc:824] T 00000000000000000000000000000000 P f27b9fed030148f1a1e894df564d965d: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:48.494233 20769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f27b9fed030148f1a1e894df564d965d: No bootstrap required, opened a new log
I20260928 15:01:48.495021 31904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:01:48.503441 20769 raft_consensus.cc:348] T 00000000000000000000000000000000 P f27b9fed030148f1a1e894df564d965d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f27b9fed030148f1a1e894df564d965d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35419 } }
I20260928 15:01:48.504070 20769 raft_consensus.cc:374] T 00000000000000000000000000000000 P f27b9fed030148f1a1e894df564d965d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:48.504197 20769 raft_consensus.cc:729] T 00000000000000000000000000000000 P f27b9fed030148f1a1e894df564d965d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f27b9fed030148f1a1e894df564d965d, State: Initialized, Role: FOLLOWER
I20260928 15:01:48.504930 20769 consensus_queue.cc:260] T 00000000000000000000000000000000 P f27b9fed030148f1a1e894df564d965d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f27b9fed030148f1a1e894df564d965d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35419 } }
I20260928 15:01:48.505244 20769 raft_consensus.cc:388] T 00000000000000000000000000000000 P f27b9fed030148f1a1e894df564d965d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:01:48.505390 20769 raft_consensus.cc:482] T 00000000000000000000000000000000 P f27b9fed030148f1a1e894df564d965d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:01:48.505594 20769 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f27b9fed030148f1a1e894df564d965d [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:48.506778 20772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:48.499579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:44365 (local address 127.0.0.1:35419)
0928 15:01:48.499808 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:48.499814 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:48.500107 (+ 293us) server_negotiation.cc:407] Connection header received
0928 15:01:48.500282 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:48.500289 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:48.500356 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:48.500460 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:48.501064 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:48.501856 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:48.503772 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:48.504474 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:48.504660 (+ 186us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:48.505943 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:48.505947 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:48.505951 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:01:48.505968 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:48.506023 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:01:48.506039 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:48.506122 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:48.506349 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 15:01:48.506409 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 15:01:48.508381 20769 raft_consensus.cc:504] T 00000000000000000000000000000000 P f27b9fed030148f1a1e894df564d965d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f27b9fed030148f1a1e894df564d965d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35419 } }
I20260928 15:01:48.508929 20769 leader_election.cc:302] T 00000000000000000000000000000000 P f27b9fed030148f1a1e894df564d965d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f27b9fed030148f1a1e894df564d965d; no voters:
I20260928 15:01:48.509614 20769 leader_election.cc:288] T 00000000000000000000000000000000 P f27b9fed030148f1a1e894df564d965d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:01:48.509743 20779 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f27b9fed030148f1a1e894df564d965d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:01:48.510511 20779 raft_consensus.cc:686] T 00000000000000000000000000000000 P f27b9fed030148f1a1e894df564d965d [term 1 LEADER]: Becoming Leader. State: Replica: f27b9fed030148f1a1e894df564d965d, State: Running, Role: LEADER
I20260928 15:01:48.511163 20779 consensus_queue.cc:237] T 00000000000000000000000000000000 P f27b9fed030148f1a1e894df564d965d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f27b9fed030148f1a1e894df564d965d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35419 } }
I20260928 15:01:48.512143 20769 sys_catalog.cc:564] T 00000000000000000000000000000000 P f27b9fed030148f1a1e894df564d965d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:01:48.514339 20781 sys_catalog.cc:455] T 00000000000000000000000000000000 P f27b9fed030148f1a1e894df564d965d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f27b9fed030148f1a1e894df564d965d. Latest consensus state: current_term: 1 leader_uuid: "f27b9fed030148f1a1e894df564d965d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f27b9fed030148f1a1e894df564d965d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35419 } } }
I20260928 15:01:48.514825 20781 sys_catalog.cc:458] T 00000000000000000000000000000000 P f27b9fed030148f1a1e894df564d965d [sys.catalog]: This master's current role is: LEADER
I20260928 15:01:48.516142 20786 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:01:48.516100 20780 sys_catalog.cc:455] T 00000000000000000000000000000000 P f27b9fed030148f1a1e894df564d965d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f27b9fed030148f1a1e894df564d965d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f27b9fed030148f1a1e894df564d965d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35419 } } }
I20260928 15:01:48.516542 20780 sys_catalog.cc:458] T 00000000000000000000000000000000 P f27b9fed030148f1a1e894df564d965d [sys.catalog]: This master's current role is: LEADER
I20260928 15:01:48.521284 20786 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:01:48.530900 20786 catalog_manager.cc:1347] Generated new cluster ID: 2e820426bd45449583f743893aeafbb2
I20260928 15:01:48.531028 20786 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:01:48.547185 20786 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:01:48.549050 20786 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:01:48.560132 20786 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f27b9fed030148f1a1e894df564d965d: Generated new TSK 0
I20260928 15:01:48.561030 20786 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:01:48.572744 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 20702
W20260928 15:01:48.574028 20764 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.152:46001
W20260928 15:01:48.574882 31904 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 20702: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:01:48.575057 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 20702
W20260928 15:01:48.592897 20687 connection.cc:535] client connection to 127.0.0.1:35419 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35419): Connection reset by peer (error 104)
2026-09-28T15:01:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1969 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607365362981-31904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:01:48 dist-test-slave-j968 krb5kdc[20800](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:01:48 dist-test-slave-j968 krb5kdc[20800](info): set up 2 sockets
Sep 28 15:01:48 dist-test-slave-j968 krb5kdc[20800](info): commencing operation
krb5kdc: starting...
W20260928 15:01:49.777585 31904 mini_kdc.cc:121] Time spent starting KDC: real 1.165s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Sep 28 15:01:49 dist-test-slave-j968 krb5kdc[20800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:01:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:01:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607365362981-31904-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607365362981-31904-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:01:50.055413 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607365362981-31904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607365362981-31904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607365362981-31904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33971
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607365362981-31904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:01:50.398239 20813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:50.398532 20813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:50.398743 20813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:50.406117 20813 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:01:50.406240 20813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:50.406360 20813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:50.406476 20813 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:01:50.406558 20813 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:01:50.416879 20813 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607365362981-31904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607365362981-31904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607365362981-31904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607365362981-31904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:01:50.418241 20813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:50.419595 20813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:50.432145 20818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:50.432096 20819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:50.432060 20821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:50.432915 20813 server_base.cc:1034] running on GCE node
I20260928 15:01:50.433709 20813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:50.435154 20813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:50.436393 20813 hybrid_clock.cc:648] HybridClock initialized: now 1790607710436342 us; error 36 us; skew 500 ppm
I20260928 15:01:50.436762 20813 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:01:50 dist-test-slave-j968 krb5kdc[20800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607710, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:01:50.445137 20813 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:01:50.446959 20813 webserver.cc:460] Webserver started at http://127.0.0.1:38475/ using document root <none> and password file <none>
I20260928 15:01:50.447718 20813 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:50.447924 20813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:50.448274 20813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:50.451467 20813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607365362981-31904-0/minicluster-data/master-0/data/instance:
uuid: "de3d35a667094317a8e5970f7d0242d8"
format_stamp: "Formatted at 2026-09-28 15:01:50 on dist-test-slave-j968"
server_key: "acf56baad201315763de6fed83fb0e1d"
server_key_iv: "8e83c680c78b21ad2e940527bf2fd446"
server_key_version: "encryptionkey@0"
I20260928 15:01:50.452319 20813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607365362981-31904-0/minicluster-data/master-0/wal/instance:
uuid: "de3d35a667094317a8e5970f7d0242d8"
format_stamp: "Formatted at 2026-09-28 15:01:50 on dist-test-slave-j968"
server_key: "acf56baad201315763de6fed83fb0e1d"
server_key_iv: "8e83c680c78b21ad2e940527bf2fd446"
server_key_version: "encryptionkey@0"
I20260928 15:01:50.457571 20813 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260928 15:01:50.461180 20828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:50.462523 20813 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260928 15:01:50.462711 20813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607365362981-31904-0/minicluster-data/master-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607365362981-31904-0/minicluster-data/master-0/wal
uuid: "de3d35a667094317a8e5970f7d0242d8"
format_stamp: "Formatted at 2026-09-28 15:01:50 on dist-test-slave-j968"
server_key: "acf56baad201315763de6fed83fb0e1d"
server_key_iv: "8e83c680c78b21ad2e940527bf2fd446"
server_key_version: "encryptionkey@0"
I20260928 15:01:50.462939 20813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607365362981-31904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607365362981-31904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607365362981-31904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:50.468808 20813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:50.472811 20813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:50.473162 20813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:50.492074 20813 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33971
I20260928 15:01:50.492159 20879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33971 every 8 connection(s)
I20260928 15:01:50.493494 20813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
I20260928 15:01:50.497323 20880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:50.501911 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 20813
I20260928 15:01:50.502101 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607365362981-31904-0/minicluster-data/master-0/wal/instance
I20260928 15:01:50.502636 31904 external_mini_cluster.cc:1428] Setting key 86df4180f82b1b7d49f445c7a9d12437
I20260928 15:01:50.509562 20880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de3d35a667094317a8e5970f7d0242d8: Bootstrap starting.
I20260928 15:01:50.511552 20883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:50.505150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:60579 (local address 127.0.0.1:33971)
0928 15:01:50.505944 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:50.505956 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:50.506004 (+ 48us) server_negotiation.cc:407] Connection header received
0928 15:01:50.508032 (+ 2028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:50.508057 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:50.508412 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:50.508796 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:50.509843 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:50.509850 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:50.509863 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:01:50.509876 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:50.509943 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:01:50.509963 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:50.510206 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:50.510347 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 15:01:50.510534 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":328,"threads_started":1}
I20260928 15:01:50.513447 20880 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de3d35a667094317a8e5970f7d0242d8: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:50.514815 20880 log.cc:824] T 00000000000000000000000000000000 P de3d35a667094317a8e5970f7d0242d8: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:50.515331 31904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:01:50.517666 20880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de3d35a667094317a8e5970f7d0242d8: No bootstrap required, opened a new log
I20260928 15:01:50.522310 20883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:50.519751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:42047 (local address 127.0.0.1:33971)
0928 15:01:50.520022 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:50.520029 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:50.520381 (+ 352us) server_negotiation.cc:407] Connection header received
0928 15:01:50.520554 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:50.520558 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:50.520609 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:50.520691 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:50.521633 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:50.521636 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:50.521639 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:01:50.521656 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:50.521691 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:01:50.521721 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:50.521806 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:50.521954 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 15:01:50.522005 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 15:01:50.526562 20880 raft_consensus.cc:348] T 00000000000000000000000000000000 P de3d35a667094317a8e5970f7d0242d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3d35a667094317a8e5970f7d0242d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33971 } }
I20260928 15:01:50.526952 20880 raft_consensus.cc:374] T 00000000000000000000000000000000 P de3d35a667094317a8e5970f7d0242d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:50.527067 20880 raft_consensus.cc:729] T 00000000000000000000000000000000 P de3d35a667094317a8e5970f7d0242d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de3d35a667094317a8e5970f7d0242d8, State: Initialized, Role: FOLLOWER
I20260928 15:01:50.527657 20880 consensus_queue.cc:260] T 00000000000000000000000000000000 P de3d35a667094317a8e5970f7d0242d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3d35a667094317a8e5970f7d0242d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33971 } }
I20260928 15:01:50.527945 20880 raft_consensus.cc:388] T 00000000000000000000000000000000 P de3d35a667094317a8e5970f7d0242d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:01:50.528105 20880 raft_consensus.cc:482] T 00000000000000000000000000000000 P de3d35a667094317a8e5970f7d0242d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:01:50.528294 20880 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de3d35a667094317a8e5970f7d0242d8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:50.530515 20880 raft_consensus.cc:504] T 00000000000000000000000000000000 P de3d35a667094317a8e5970f7d0242d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3d35a667094317a8e5970f7d0242d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33971 } }
I20260928 15:01:50.531028 20880 leader_election.cc:302] T 00000000000000000000000000000000 P de3d35a667094317a8e5970f7d0242d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de3d35a667094317a8e5970f7d0242d8; no voters:
I20260928 15:01:50.531610 20880 leader_election.cc:288] T 00000000000000000000000000000000 P de3d35a667094317a8e5970f7d0242d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:01:50.531749 20890 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de3d35a667094317a8e5970f7d0242d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:01:50.532491 20890 raft_consensus.cc:686] T 00000000000000000000000000000000 P de3d35a667094317a8e5970f7d0242d8 [term 1 LEADER]: Becoming Leader. State: Replica: de3d35a667094317a8e5970f7d0242d8, State: Running, Role: LEADER
I20260928 15:01:50.533138 20890 consensus_queue.cc:237] T 00000000000000000000000000000000 P de3d35a667094317a8e5970f7d0242d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3d35a667094317a8e5970f7d0242d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33971 } }
I20260928 15:01:50.533931 20880 sys_catalog.cc:564] T 00000000000000000000000000000000 P de3d35a667094317a8e5970f7d0242d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:01:50.536460 20891 sys_catalog.cc:455] T 00000000000000000000000000000000 P de3d35a667094317a8e5970f7d0242d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de3d35a667094317a8e5970f7d0242d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3d35a667094317a8e5970f7d0242d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33971 } } }
I20260928 15:01:50.536890 20891 sys_catalog.cc:458] T 00000000000000000000000000000000 P de3d35a667094317a8e5970f7d0242d8 [sys.catalog]: This master's current role is: LEADER
I20260928 15:01:50.536427 20892 sys_catalog.cc:455] T 00000000000000000000000000000000 P de3d35a667094317a8e5970f7d0242d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de3d35a667094317a8e5970f7d0242d8. Latest consensus state: current_term: 1 leader_uuid: "de3d35a667094317a8e5970f7d0242d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3d35a667094317a8e5970f7d0242d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33971 } } }
I20260928 15:01:50.537213 20892 sys_catalog.cc:458] T 00000000000000000000000000000000 P de3d35a667094317a8e5970f7d0242d8 [sys.catalog]: This master's current role is: LEADER
I20260928 15:01:50.540974 20896 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:01:50.545650 20896 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:01:50.554054 20896 catalog_manager.cc:1347] Generated new cluster ID: 79ff2a5c94034c51851010e30c6ba6c5
I20260928 15:01:50.554180 20896 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:01:50.570046 20896 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:01:50.570571 20896 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:01:50.580657 20896 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de3d35a667094317a8e5970f7d0242d8: Generated new TSK 0
I20260928 15:01:50.581871 20896 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:01:50.587492 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 20813
W20260928 15:01:50.588971 20875 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.152:60579
W20260928 15:01:50.589766 31904 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 20813: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:01:50.589967 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 20813
2026-09-28T15:01:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2015 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607365362981-31904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:01:50 dist-test-slave-j968 krb5kdc[20911](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:01:50 dist-test-slave-j968 krb5kdc[20911](info): set up 2 sockets
Sep 28 15:01:50 dist-test-slave-j968 krb5kdc[20911](info): commencing operation
krb5kdc: starting...
W20260928 15:01:51.697232 31904 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Sep 28 15:01:51 dist-test-slave-j968 krb5kdc[20911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:01:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:01:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607365362981-31904-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607365362981-31904-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:01:51.967751 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607365362981-31904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607365362981-31904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607365362981-31904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43201
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607365362981-31904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:01:52.323688 20924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:52.324143 20924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:52.324317 20924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:52.331823 20924 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:01:52.332031 20924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:52.332084 20924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:52.332150 20924 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:01:52.332194 20924 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:01:52.342744 20924 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607365362981-31904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607365362981-31904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43201
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607365362981-31904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607365362981-31904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:01:52.344197 20924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:52.345504 20924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:52.358206 20930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:52.358397 20932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:52.358940 20924 server_base.cc:1034] running on GCE node
W20260928 15:01:52.359057 20929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:52.360294 20924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:52.362946 20924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:52.364220 20924 hybrid_clock.cc:648] HybridClock initialized: now 1790607712364156 us; error 55 us; skew 500 ppm
I20260928 15:01:52.364742 20924 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:01:52 dist-test-slave-j968 krb5kdc[20911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607712, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:01:52.368369 20924 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:01:52.369797 20924 webserver.cc:460] Webserver started at http://127.0.0.1:39703/ using document root <none> and password file <none>
I20260928 15:01:52.370447 20924 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:52.370548 20924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:52.370801 20924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:52.373991 20924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607365362981-31904-0/minicluster-data/master-0/data/instance:
uuid: "79aa6d61686e448abadd3ac79687e31f"
format_stamp: "Formatted at 2026-09-28 15:01:52 on dist-test-slave-j968"
server_key: "5ae0017a48cae3fa686cbc1fa5470df0"
server_key_iv: "5f6c83167d621a5a408aaa49d27636ae"
server_key_version: "encryptionkey@0"
I20260928 15:01:52.374711 20924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607365362981-31904-0/minicluster-data/master-0/wal/instance:
uuid: "79aa6d61686e448abadd3ac79687e31f"
format_stamp: "Formatted at 2026-09-28 15:01:52 on dist-test-slave-j968"
server_key: "5ae0017a48cae3fa686cbc1fa5470df0"
server_key_iv: "5f6c83167d621a5a408aaa49d27636ae"
server_key_version: "encryptionkey@0"
I20260928 15:01:52.380044 20924 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260928 15:01:52.383523 20939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:52.384989 20924 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:01:52.385210 20924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607365362981-31904-0/minicluster-data/master-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607365362981-31904-0/minicluster-data/master-0/wal
uuid: "79aa6d61686e448abadd3ac79687e31f"
format_stamp: "Formatted at 2026-09-28 15:01:52 on dist-test-slave-j968"
server_key: "5ae0017a48cae3fa686cbc1fa5470df0"
server_key_iv: "5f6c83167d621a5a408aaa49d27636ae"
server_key_version: "encryptionkey@0"
I20260928 15:01:52.385416 20924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607365362981-31904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607365362981-31904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607365362981-31904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:52.408468 20924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:52.413132 20924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:52.413455 20924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:52.433054 20924 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43201
I20260928 15:01:52.433099 20990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43201 every 8 connection(s)
I20260928 15:01:52.434686 20924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
I20260928 15:01:52.434993 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 20924
I20260928 15:01:52.435192 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607365362981-31904-0/minicluster-data/master-0/wal/instance
I20260928 15:01:52.435686 31904 external_mini_cluster.cc:1428] Setting key 70ca2b5062e0c9d0424696358f6d27da
I20260928 15:01:52.439419 20991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 15:01:52 dist-test-slave-j968 krb5kdc[20911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 15:01:52.452888 20991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79aa6d61686e448abadd3ac79687e31f: Bootstrap starting.
I20260928 15:01:52.456295 20991 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79aa6d61686e448abadd3ac79687e31f: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:52.457422 20991 log.cc:824] T 00000000000000000000000000000000 P 79aa6d61686e448abadd3ac79687e31f: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:52.459533 20994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:52.437759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:57157 (local address 127.0.0.1:43201)
0928 15:01:52.438540 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:52.438552 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:52.438593 (+ 41us) server_negotiation.cc:407] Connection header received
0928 15:01:52.440450 (+ 1857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:52.440473 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:52.440869 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:52.441184 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:52.443058 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:52.444452 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:52.446282 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:52.446981 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:52.447125 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:52.451848 (+ 4723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:52.451856 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:52.451931 (+ 75us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:52.451981 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:52.455164 (+ 3183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:52.456090 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:52.456097 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:52.456104 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:52.456278 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:52.456773 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:52.456777 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:52.456780 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:52.457306 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:52.457675 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:52.458202 (+ 527us) server_negotiation.cc:299] Negotiation successful
0928 15:01:52.458530 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":354,"threads_started":1}
I20260928 15:01:52.460706 20991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79aa6d61686e448abadd3ac79687e31f: No bootstrap required, opened a new log
I20260928 15:01:52.463554 31904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:01:52.469746 20991 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79aa6d61686e448abadd3ac79687e31f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79aa6d61686e448abadd3ac79687e31f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43201 } }
I20260928 15:01:52.470322 20991 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79aa6d61686e448abadd3ac79687e31f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:52.470444 20991 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79aa6d61686e448abadd3ac79687e31f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79aa6d61686e448abadd3ac79687e31f, State: Initialized, Role: FOLLOWER
I20260928 15:01:52.471190 20991 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79aa6d61686e448abadd3ac79687e31f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79aa6d61686e448abadd3ac79687e31f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43201 } }
I20260928 15:01:52.471469 20991 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79aa6d61686e448abadd3ac79687e31f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:01:52.471621 20991 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79aa6d61686e448abadd3ac79687e31f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:01:52.471817 20991 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79aa6d61686e448abadd3ac79687e31f [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:52.474965 20991 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79aa6d61686e448abadd3ac79687e31f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79aa6d61686e448abadd3ac79687e31f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43201 } }
I20260928 15:01:52.475662 20991 leader_election.cc:302] T 00000000000000000000000000000000 P 79aa6d61686e448abadd3ac79687e31f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79aa6d61686e448abadd3ac79687e31f; no voters:
I20260928 15:01:52.476408 20991 leader_election.cc:288] T 00000000000000000000000000000000 P 79aa6d61686e448abadd3ac79687e31f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:01:52.476521 21001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79aa6d61686e448abadd3ac79687e31f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:01:52.477924 21001 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79aa6d61686e448abadd3ac79687e31f [term 1 LEADER]: Becoming Leader. State: Replica: 79aa6d61686e448abadd3ac79687e31f, State: Running, Role: LEADER
I20260928 15:01:52.478576 21001 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79aa6d61686e448abadd3ac79687e31f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79aa6d61686e448abadd3ac79687e31f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43201 } }
I20260928 15:01:52.479104 20991 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79aa6d61686e448abadd3ac79687e31f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:01:52.481747 21003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79aa6d61686e448abadd3ac79687e31f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79aa6d61686e448abadd3ac79687e31f. Latest consensus state: current_term: 1 leader_uuid: "79aa6d61686e448abadd3ac79687e31f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79aa6d61686e448abadd3ac79687e31f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43201 } } }
I20260928 15:01:52.482158 21003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79aa6d61686e448abadd3ac79687e31f [sys.catalog]: This master's current role is: LEADER
I20260928 15:01:52.482681 21002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79aa6d61686e448abadd3ac79687e31f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79aa6d61686e448abadd3ac79687e31f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79aa6d61686e448abadd3ac79687e31f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43201 } } }
I20260928 15:01:52.482985 21002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79aa6d61686e448abadd3ac79687e31f [sys.catalog]: This master's current role is: LEADER
I20260928 15:01:52.488354 20994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:52.468066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:55895 (local address 127.0.0.1:43201)
0928 15:01:52.468317 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:52.468327 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:52.468693 (+ 366us) server_negotiation.cc:407] Connection header received
0928 15:01:52.468883 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:52.468887 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:52.468973 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:52.469049 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:52.471035 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:52.472090 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:52.473871 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:52.474326 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:52.474420 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:52.477976 (+ 3556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:52.477984 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:52.477988 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:52.478040 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:52.482594 (+ 4554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:52.484062 (+ 1468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:52.484071 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:52.484078 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:52.484302 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:52.485040 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:52.485048 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:52.485054 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:52.485561 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:52.487287 (+ 1726us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:52.487519 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 15:01:52.487767 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 15:01:52.501122 21006 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:01:52.507421 21006 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:01:52.516947 21006 catalog_manager.cc:1347] Generated new cluster ID: a655839ce70740ef9a871826e8c1f9ed
I20260928 15:01:52.517078 21006 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:01:52.528014 21006 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:01:52.529901 21006 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:01:52.546795 21006 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79aa6d61686e448abadd3ac79687e31f: Generated new TSK 0
I20260928 15:01:52.547950 21006 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:01:52.561587 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 20924
I20260928 15:01:52.563581 20986 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:01:52.658411 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 20924
W20260928 15:01:52.676623 20909 connection.cc:535] client connection to 127.0.0.1:43201 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43201): Connection reset by peer (error 104)
2026-09-28T15:01:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2068 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607365362981-31904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:01:52 dist-test-slave-j968 krb5kdc[21024](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:01:52 dist-test-slave-j968 krb5kdc[21024](info): set up 2 sockets
Sep 28 15:01:52 dist-test-slave-j968 krb5kdc[21024](info): commencing operation
krb5kdc: starting...
W20260928 15:01:53.777379 31904 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Sep 28 15:01:53 dist-test-slave-j968 krb5kdc[21024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:01:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:01:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607365362981-31904-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607365362981-31904-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:01:54.053380 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607365362981-31904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607365362981-31904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607365362981-31904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39113
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607365362981-31904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39113
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:01:54.401404 21037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:54.401737 21037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:54.401834 21037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:54.409361 21037 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:01:54.409483 21037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:54.409529 21037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:54.409590 21037 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:01:54.409633 21037 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:01:54.419797 21037 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607365362981-31904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607365362981-31904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39113
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607365362981-31904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607365362981-31904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:01:54.421240 21037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:54.422582 21037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:54.435027 21042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:54.434995 21043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:54.435068 21045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:54.435967 21037 server_base.cc:1034] running on GCE node
I20260928 15:01:54.436767 21037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:54.438426 21037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:54.439705 21037 hybrid_clock.cc:648] HybridClock initialized: now 1790607714439642 us; error 44 us; skew 500 ppm
I20260928 15:01:54.440156 21037 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:01:54 dist-test-slave-j968 krb5kdc[21024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:01:54.443572 21037 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:01:54.445288 21037 webserver.cc:460] Webserver started at http://127.0.0.1:40509/ using document root <none> and password file <none>
I20260928 15:01:54.446033 21037 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:54.446192 21037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:54.446517 21037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:54.449791 21037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607365362981-31904-0/minicluster-data/master-0/data/instance:
uuid: "056d42f201334d96bc4afff2f36a54df"
format_stamp: "Formatted at 2026-09-28 15:01:54 on dist-test-slave-j968"
server_key: "aa8cc8fce849aa8087bba28e25af7c6a"
server_key_iv: "8596cc5d1595aa17fef1bbcc9eb0c470"
server_key_version: "encryptionkey@0"
I20260928 15:01:54.450636 21037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607365362981-31904-0/minicluster-data/master-0/wal/instance:
uuid: "056d42f201334d96bc4afff2f36a54df"
format_stamp: "Formatted at 2026-09-28 15:01:54 on dist-test-slave-j968"
server_key: "aa8cc8fce849aa8087bba28e25af7c6a"
server_key_iv: "8596cc5d1595aa17fef1bbcc9eb0c470"
server_key_version: "encryptionkey@0"
I20260928 15:01:54.456436 21037 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260928 15:01:54.460397 21052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:54.461853 21037 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:01:54.462069 21037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607365362981-31904-0/minicluster-data/master-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607365362981-31904-0/minicluster-data/master-0/wal
uuid: "056d42f201334d96bc4afff2f36a54df"
format_stamp: "Formatted at 2026-09-28 15:01:54 on dist-test-slave-j968"
server_key: "aa8cc8fce849aa8087bba28e25af7c6a"
server_key_iv: "8596cc5d1595aa17fef1bbcc9eb0c470"
server_key_version: "encryptionkey@0"
I20260928 15:01:54.462288 21037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607365362981-31904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607365362981-31904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607365362981-31904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:54.483690 21037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:54.488415 21037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:54.488725 21037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:54.508558 21037 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39113
I20260928 15:01:54.508595 21103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39113 every 8 connection(s)
I20260928 15:01:54.509966 21037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
I20260928 15:01:54.514117 21104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:54.519574 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 21037
I20260928 15:01:54.519753 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607365362981-31904-0/minicluster-data/master-0/wal/instance
I20260928 15:01:54.520313 31904 external_mini_cluster.cc:1428] Setting key 80a6e2d6c26380aaad9188a40f855640
I20260928 15:01:54.526073 21104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 056d42f201334d96bc4afff2f36a54df: Bootstrap starting.
I20260928 15:01:54.529907 21104 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 056d42f201334d96bc4afff2f36a54df: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:54.531213 21104 log.cc:824] T 00000000000000000000000000000000 P 056d42f201334d96bc4afff2f36a54df: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:54.534587 21104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 056d42f201334d96bc4afff2f36a54df: No bootstrap required, opened a new log
I20260928 15:01:54.534770 21107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:54.522239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:45251 (local address 127.0.0.1:39113)
0928 15:01:54.523147 (+ 908us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:54.523162 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:54.523206 (+ 44us) server_negotiation.cc:407] Connection header received
0928 15:01:54.525085 (+ 1879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:54.525111 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:54.525547 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:54.525877 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:54.526523 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:54.527852 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:54.529740 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:54.530555 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:54.530691 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:54.532145 (+ 1454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:54.532156 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:54.532177 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:01:54.532943 (+ 766us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:54.533017 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:01:54.533042 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:54.533164 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:54.533512 (+ 348us) server_negotiation.cc:299] Negotiation successful
0928 15:01:54.533723 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":406,"threads_started":1}
I20260928 15:01:54.538667 31904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:01:54.544595 21104 raft_consensus.cc:348] T 00000000000000000000000000000000 P 056d42f201334d96bc4afff2f36a54df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "056d42f201334d96bc4afff2f36a54df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39113 } }
I20260928 15:01:54.545032 21104 raft_consensus.cc:374] T 00000000000000000000000000000000 P 056d42f201334d96bc4afff2f36a54df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:54.545166 21104 raft_consensus.cc:729] T 00000000000000000000000000000000 P 056d42f201334d96bc4afff2f36a54df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 056d42f201334d96bc4afff2f36a54df, State: Initialized, Role: FOLLOWER
I20260928 15:01:54.545802 21104 consensus_queue.cc:260] T 00000000000000000000000000000000 P 056d42f201334d96bc4afff2f36a54df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "056d42f201334d96bc4afff2f36a54df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39113 } }
I20260928 15:01:54.546082 21104 raft_consensus.cc:388] T 00000000000000000000000000000000 P 056d42f201334d96bc4afff2f36a54df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:01:54.546267 21104 raft_consensus.cc:482] T 00000000000000000000000000000000 P 056d42f201334d96bc4afff2f36a54df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:01:54.546464 21104 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 056d42f201334d96bc4afff2f36a54df [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:54.548872 21104 raft_consensus.cc:504] T 00000000000000000000000000000000 P 056d42f201334d96bc4afff2f36a54df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "056d42f201334d96bc4afff2f36a54df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39113 } }
I20260928 15:01:54.549443 21104 leader_election.cc:302] T 00000000000000000000000000000000 P 056d42f201334d96bc4afff2f36a54df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 056d42f201334d96bc4afff2f36a54df; no voters:
I20260928 15:01:54.550148 21104 leader_election.cc:288] T 00000000000000000000000000000000 P 056d42f201334d96bc4afff2f36a54df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:01:54.550259 21114 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 056d42f201334d96bc4afff2f36a54df [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:01:54.551525 21114 raft_consensus.cc:686] T 00000000000000000000000000000000 P 056d42f201334d96bc4afff2f36a54df [term 1 LEADER]: Becoming Leader. State: Replica: 056d42f201334d96bc4afff2f36a54df, State: Running, Role: LEADER
I20260928 15:01:54.552134 21107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:54.544293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:40753 (local address 127.0.0.1:39113)
0928 15:01:54.544524 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:54.544532 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:54.544958 (+ 426us) server_negotiation.cc:407] Connection header received
0928 15:01:54.545179 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:54.545185 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:54.545276 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:54.545387 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:54.546111 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:54.547121 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:54.548756 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:54.549359 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:54.549514 (+ 155us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:54.550980 (+ 1466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:54.550989 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:54.550994 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:01:54.551030 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:54.551101 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:01:54.551122 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:54.551234 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:54.551680 (+ 446us) server_negotiation.cc:299] Negotiation successful
0928 15:01:54.551740 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 15:01:54.552393 21114 consensus_queue.cc:237] T 00000000000000000000000000000000 P 056d42f201334d96bc4afff2f36a54df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "056d42f201334d96bc4afff2f36a54df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39113 } }
I20260928 15:01:54.552764 21104 sys_catalog.cc:564] T 00000000000000000000000000000000 P 056d42f201334d96bc4afff2f36a54df [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:01:54.555688 21115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 056d42f201334d96bc4afff2f36a54df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "056d42f201334d96bc4afff2f36a54df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "056d42f201334d96bc4afff2f36a54df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39113 } } }
I20260928 15:01:54.556170 21115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 056d42f201334d96bc4afff2f36a54df [sys.catalog]: This master's current role is: LEADER
I20260928 15:01:54.556372 21116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 056d42f201334d96bc4afff2f36a54df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 056d42f201334d96bc4afff2f36a54df. Latest consensus state: current_term: 1 leader_uuid: "056d42f201334d96bc4afff2f36a54df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "056d42f201334d96bc4afff2f36a54df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39113 } } }
I20260928 15:01:54.556636 21116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 056d42f201334d96bc4afff2f36a54df [sys.catalog]: This master's current role is: LEADER
I20260928 15:01:54.558859 21123 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:01:54.564778 21123 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:01:54.575538 21123 catalog_manager.cc:1347] Generated new cluster ID: 5ec5d76a3b2e4dc78fd0d6f15886f382
I20260928 15:01:54.575685 21123 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:01:54.595781 21123 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:01:54.597162 21123 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:01:54.607283 21123 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 056d42f201334d96bc4afff2f36a54df: Generated new TSK 0
I20260928 15:01:54.608234 21123 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:01:54.620651 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 21037
W20260928 15:01:54.621884 21099 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.152:45251
W20260928 15:01:54.622861 31904 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 21037: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:01:54.623031 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 21037
W20260928 15:01:54.638351 21022 connection.cc:535] client connection to 127.0.0.1:39113 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39113): Connection reset by peer (error 104)
2026-09-28T15:01:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1961 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607365362981-31904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:01:54 dist-test-slave-j968 krb5kdc[21135](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:01:54 dist-test-slave-j968 krb5kdc[21135](info): set up 2 sockets
Sep 28 15:01:54 dist-test-slave-j968 krb5kdc[21135](info): commencing operation
krb5kdc: starting...
W20260928 15:01:55.741351 31904 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Sep 28 15:01:55 dist-test-slave-j968 krb5kdc[21135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:01:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:01:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607365362981-31904-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607365362981-31904-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:01:56.013062 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607365362981-31904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607365362981-31904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607365362981-31904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41129
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607365362981-31904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:01:56.368537 21148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:56.368840 21148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:56.369062 21148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:56.376399 21148 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:01:56.376549 21148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:56.376653 21148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:56.376766 21148 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:01:56.376832 21148 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:01:56.386834 21148 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607365362981-31904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607365362981-31904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41129
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607365362981-31904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607365362981-31904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:01:56.388229 21148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:56.389569 21148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:56.401554 21153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:56.401608 21156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:56.401523 21154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:56.402657 21148 server_base.cc:1034] running on GCE node
I20260928 15:01:56.403515 21148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:56.405198 21148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:56.406451 21148 hybrid_clock.cc:648] HybridClock initialized: now 1790607716406406 us; error 35 us; skew 500 ppm
I20260928 15:01:56.406814 21148 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:01:56 dist-test-slave-j968 krb5kdc[21135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607716, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:01:56.409981 21148 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:01:56.411526 21148 webserver.cc:460] Webserver started at http://127.0.0.1:45551/ using document root <none> and password file <none>
I20260928 15:01:56.412312 21148 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:56.412462 21148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:56.412771 21148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:56.415772 21148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607365362981-31904-0/minicluster-data/master-0/data/instance:
uuid: "8130a5784e364f5cafcd8d4d7a2f2210"
format_stamp: "Formatted at 2026-09-28 15:01:56 on dist-test-slave-j968"
server_key: "a852301bc768c11f500f5823f112d0c7"
server_key_iv: "27eab0aa877bde2800e92f2d5c7ccfe9"
server_key_version: "encryptionkey@0"
I20260928 15:01:56.416620 21148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607365362981-31904-0/minicluster-data/master-0/wal/instance:
uuid: "8130a5784e364f5cafcd8d4d7a2f2210"
format_stamp: "Formatted at 2026-09-28 15:01:56 on dist-test-slave-j968"
server_key: "a852301bc768c11f500f5823f112d0c7"
server_key_iv: "27eab0aa877bde2800e92f2d5c7ccfe9"
server_key_version: "encryptionkey@0"
I20260928 15:01:56.421909 21148 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260928 15:01:56.425539 21163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:56.426951 21148 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 15:01:56.427148 21148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607365362981-31904-0/minicluster-data/master-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607365362981-31904-0/minicluster-data/master-0/wal
uuid: "8130a5784e364f5cafcd8d4d7a2f2210"
format_stamp: "Formatted at 2026-09-28 15:01:56 on dist-test-slave-j968"
server_key: "a852301bc768c11f500f5823f112d0c7"
server_key_iv: "27eab0aa877bde2800e92f2d5c7ccfe9"
server_key_version: "encryptionkey@0"
I20260928 15:01:56.427364 21148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607365362981-31904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607365362981-31904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607365362981-31904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:56.433362 21148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:56.437579 21148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:56.437896 21148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:56.457077 21148 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41129
I20260928 15:01:56.457183 21214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41129 every 8 connection(s)
I20260928 15:01:56.458984 21148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
I20260928 15:01:56.462934 21215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:56.466652 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 21148
I20260928 15:01:56.466842 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607365362981-31904-0/minicluster-data/master-0/wal/instance
I20260928 15:01:56.467409 31904 external_mini_cluster.cc:1428] Setting key 82781a31ed42eb357a257209db38faed
I20260928 15:01:56.475423 21218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:56.469403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:55017 (local address 127.0.0.1:41129)
0928 15:01:56.470162 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:56.470174 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:56.470215 (+ 41us) server_negotiation.cc:407] Connection header received
0928 15:01:56.471849 (+ 1634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:56.471949 (+ 100us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:56.472337 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:56.472753 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:56.473823 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:56.473829 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:56.473842 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:01:56.473858 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:56.473920 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:01:56.473939 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:56.474168 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:56.474312 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 15:01:56.474504 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":306,"threads_started":1}
I20260928 15:01:56.475687 21215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8130a5784e364f5cafcd8d4d7a2f2210: Bootstrap starting.
I20260928 15:01:56.479039 21215 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8130a5784e364f5cafcd8d4d7a2f2210: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:56.479177 31904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:01:56.480213 21215 log.cc:824] T 00000000000000000000000000000000 P 8130a5784e364f5cafcd8d4d7a2f2210: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:56.483386 21215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8130a5784e364f5cafcd8d4d7a2f2210: No bootstrap required, opened a new log
I20260928 15:01:56.486334 21218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:56.483615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:53117 (local address 127.0.0.1:41129)
0928 15:01:56.483770 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:56.483775 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:56.484292 (+ 517us) server_negotiation.cc:407] Connection header received
0928 15:01:56.484500 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:56.484511 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:56.484559 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:56.484659 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:56.485623 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:56.485628 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:56.485637 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:01:56.485656 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:56.485696 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:01:56.485713 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:56.485837 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:56.485967 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 15:01:56.486040 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 15:01:56.491638 21215 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8130a5784e364f5cafcd8d4d7a2f2210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8130a5784e364f5cafcd8d4d7a2f2210" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41129 } }
I20260928 15:01:56.492082 21215 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8130a5784e364f5cafcd8d4d7a2f2210 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:56.492179 21215 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8130a5784e364f5cafcd8d4d7a2f2210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8130a5784e364f5cafcd8d4d7a2f2210, State: Initialized, Role: FOLLOWER
I20260928 15:01:56.492846 21215 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8130a5784e364f5cafcd8d4d7a2f2210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8130a5784e364f5cafcd8d4d7a2f2210" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41129 } }
I20260928 15:01:56.493139 21215 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8130a5784e364f5cafcd8d4d7a2f2210 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:01:56.493291 21215 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8130a5784e364f5cafcd8d4d7a2f2210 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:01:56.493479 21215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8130a5784e364f5cafcd8d4d7a2f2210 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:56.495651 21215 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8130a5784e364f5cafcd8d4d7a2f2210 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8130a5784e364f5cafcd8d4d7a2f2210" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41129 } }
I20260928 15:01:56.496212 21215 leader_election.cc:302] T 00000000000000000000000000000000 P 8130a5784e364f5cafcd8d4d7a2f2210 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8130a5784e364f5cafcd8d4d7a2f2210; no voters:
I20260928 15:01:56.496958 21215 leader_election.cc:288] T 00000000000000000000000000000000 P 8130a5784e364f5cafcd8d4d7a2f2210 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:01:56.497244 21225 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8130a5784e364f5cafcd8d4d7a2f2210 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:01:56.498057 21225 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8130a5784e364f5cafcd8d4d7a2f2210 [term 1 LEADER]: Becoming Leader. State: Replica: 8130a5784e364f5cafcd8d4d7a2f2210, State: Running, Role: LEADER
I20260928 15:01:56.498658 21225 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8130a5784e364f5cafcd8d4d7a2f2210 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8130a5784e364f5cafcd8d4d7a2f2210" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41129 } }
I20260928 15:01:56.499310 21215 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8130a5784e364f5cafcd8d4d7a2f2210 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:01:56.501922 21226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8130a5784e364f5cafcd8d4d7a2f2210 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8130a5784e364f5cafcd8d4d7a2f2210" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8130a5784e364f5cafcd8d4d7a2f2210" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41129 } } }
I20260928 15:01:56.501942 21227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8130a5784e364f5cafcd8d4d7a2f2210 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8130a5784e364f5cafcd8d4d7a2f2210. Latest consensus state: current_term: 1 leader_uuid: "8130a5784e364f5cafcd8d4d7a2f2210" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8130a5784e364f5cafcd8d4d7a2f2210" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41129 } } }
I20260928 15:01:56.503062 21226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8130a5784e364f5cafcd8d4d7a2f2210 [sys.catalog]: This master's current role is: LEADER
I20260928 15:01:56.503062 21227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8130a5784e364f5cafcd8d4d7a2f2210 [sys.catalog]: This master's current role is: LEADER
I20260928 15:01:56.504549 21234 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:01:56.511442 21234 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:01:56.522361 21234 catalog_manager.cc:1347] Generated new cluster ID: 2a418d567c984a66afc9ed3413108b4d
I20260928 15:01:56.522496 21234 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:01:56.550640 21234 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:01:56.551180 21234 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:01:56.559085 21234 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8130a5784e364f5cafcd8d4d7a2f2210: Generated new TSK 0
I20260928 15:01:56.560209 21234 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:01:56.576649 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 21148
W20260928 15:01:56.579056 21210 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.152:55017
W20260928 15:01:56.579790 31904 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 21148: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:01:56.579996 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 21148
2026-09-28T15:01:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1959 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607365362981-31904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:01:56 dist-test-slave-j968 krb5kdc[21246](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:01:56 dist-test-slave-j968 krb5kdc[21246](info): set up 2 sockets
Sep 28 15:01:56 dist-test-slave-j968 krb5kdc[21246](info): commencing operation
krb5kdc: starting...
W20260928 15:01:57.701639 31904 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.001s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Sep 28 15:01:57 dist-test-slave-j968 krb5kdc[21246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:01:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:01:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab.
Entry for principal kudu/127.31.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab.
I20260928 15:01:57.982663 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607365362981-31904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607365362981-31904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607365362981-31904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:34783
--webserver_interface=127.31.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:33741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.40.62:34783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:01:58.327684 21259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:58.328119 21259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:58.328233 21259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:58.335752 21259 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:01:58.335927 21259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:58.336004 21259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:58.336068 21259 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:01:58.336113 21259 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:01:58.346755 21259 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:33741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607365362981-31904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607365362981-31904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.40.62:34783
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:34783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607365362981-31904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:01:58.348227 21259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:58.349604 21259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:58.361975 21264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:58.362044 21265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:58.362006 21267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:58.362583 21259 server_base.cc:1034] running on GCE node
I20260928 15:01:58.363505 21259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:58.364960 21259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:58.366197 21259 hybrid_clock.cc:648] HybridClock initialized: now 1790607718366152 us; error 61 us; skew 500 ppm
I20260928 15:01:58.366571 21259 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:01:58 dist-test-slave-j968 krb5kdc[21246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607718, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:01:58.370132 21259 init.cc:377] Logged in from keytab as kudu/127.31.40.62@KRBTEST.COM (short username kudu)
I20260928 15:01:58.371737 21259 webserver.cc:460] Webserver started at http://127.31.40.62:41923/ using document root <none> and password file <none>
I20260928 15:01:58.372526 21259 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:58.372680 21259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:58.373009 21259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:58.376158 21259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607365362981-31904-0/minicluster-data/master-0/data/instance:
uuid: "0ef55f4c58864134bdb83a565341d91f"
format_stamp: "Formatted at 2026-09-28 15:01:58 on dist-test-slave-j968"
server_key: "cf81f0c7be71f0a40e21a757681bc2ee"
server_key_iv: "befaccb5cc93f6854c9bd260c0051322"
server_key_version: "encryptionkey@0"
I20260928 15:01:58.376953 21259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607365362981-31904-0/minicluster-data/master-0/wal/instance:
uuid: "0ef55f4c58864134bdb83a565341d91f"
format_stamp: "Formatted at 2026-09-28 15:01:58 on dist-test-slave-j968"
server_key: "cf81f0c7be71f0a40e21a757681bc2ee"
server_key_iv: "befaccb5cc93f6854c9bd260c0051322"
server_key_version: "encryptionkey@0"
I20260928 15:01:58.382331 21259 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260928 15:01:58.386034 21274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:58.387702 21259 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 15:01:58.387915 21259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607365362981-31904-0/minicluster-data/master-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607365362981-31904-0/minicluster-data/master-0/wal
uuid: "0ef55f4c58864134bdb83a565341d91f"
format_stamp: "Formatted at 2026-09-28 15:01:58 on dist-test-slave-j968"
server_key: "cf81f0c7be71f0a40e21a757681bc2ee"
server_key_iv: "befaccb5cc93f6854c9bd260c0051322"
server_key_version: "encryptionkey@0"
I20260928 15:01:58.388180 21259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607365362981-31904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607365362981-31904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607365362981-31904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:58.412307 21259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:58.417465 21259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:58.417819 21259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:58.437081 21259 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.62:34783
I20260928 15:01:58.437112 21325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.62:34783 every 8 connection(s)
I20260928 15:01:58.438624 21259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
I20260928 15:01:58.439160 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 21259
I20260928 15:01:58.439366 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607365362981-31904-0/minicluster-data/master-0/wal/instance
I20260928 15:01:58.440006 31904 external_mini_cluster.cc:1428] Setting key e5abdaed945bda8e240b8d7d4231e8c4
I20260928 15:01:58.443470 21326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 15:01:58 dist-test-slave-j968 krb5kdc[21246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 15:01:58.455181 21326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ef55f4c58864134bdb83a565341d91f: Bootstrap starting.
I20260928 15:01:58.459035 21326 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ef55f4c58864134bdb83a565341d91f: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:58.460331 21326 log.cc:824] T 00000000000000000000000000000000 P 0ef55f4c58864134bdb83a565341d91f: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:58.463855 21326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ef55f4c58864134bdb83a565341d91f: No bootstrap required, opened a new log
I20260928 15:01:58.464375 21329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:58.441932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32827 (local address 127.31.40.62:34783)
0928 15:01:58.442707 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:58.442719 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:58.442770 (+ 51us) server_negotiation.cc:407] Connection header received
0928 15:01:58.444631 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:58.444653 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:58.445053 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:58.445461 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:58.446965 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:58.448472 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:58.449807 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:58.450541 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:58.450742 (+ 201us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:58.455503 (+ 4761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:58.455513 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:58.455530 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:58.455577 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:58.458864 (+ 3287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:58.459947 (+ 1083us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:58.459955 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:58.459962 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:58.460143 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:58.460718 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:58.460728 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:58.460730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:58.461645 (+ 915us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:58.461976 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:58.462882 (+ 906us) server_negotiation.cc:299] Negotiation successful
0928 15:01:58.463231 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":341,"threads_started":1}
I20260928 15:01:58.468899 31904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:01:58.474822 21326 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ef55f4c58864134bdb83a565341d91f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ef55f4c58864134bdb83a565341d91f" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34783 } }
I20260928 15:01:58.475338 21326 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ef55f4c58864134bdb83a565341d91f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:58.475476 21326 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ef55f4c58864134bdb83a565341d91f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ef55f4c58864134bdb83a565341d91f, State: Initialized, Role: FOLLOWER
I20260928 15:01:58.476178 21326 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ef55f4c58864134bdb83a565341d91f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ef55f4c58864134bdb83a565341d91f" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34783 } }
I20260928 15:01:58.476421 21326 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ef55f4c58864134bdb83a565341d91f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:01:58.476576 21326 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ef55f4c58864134bdb83a565341d91f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:01:58.476752 21326 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ef55f4c58864134bdb83a565341d91f [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:58.480120 21326 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ef55f4c58864134bdb83a565341d91f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ef55f4c58864134bdb83a565341d91f" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34783 } }
I20260928 15:01:58.480752 21326 leader_election.cc:302] T 00000000000000000000000000000000 P 0ef55f4c58864134bdb83a565341d91f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ef55f4c58864134bdb83a565341d91f; no voters:
I20260928 15:01:58.481667 21326 leader_election.cc:288] T 00000000000000000000000000000000 P 0ef55f4c58864134bdb83a565341d91f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:01:58.481820 21336 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ef55f4c58864134bdb83a565341d91f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:01:58.484575 21336 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ef55f4c58864134bdb83a565341d91f [term 1 LEADER]: Becoming Leader. State: Replica: 0ef55f4c58864134bdb83a565341d91f, State: Running, Role: LEADER
I20260928 15:01:58.485339 21336 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ef55f4c58864134bdb83a565341d91f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ef55f4c58864134bdb83a565341d91f" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34783 } }
I20260928 15:01:58.486033 21326 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ef55f4c58864134bdb83a565341d91f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:01:58.493403 21329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:58.474442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59193 (local address 127.31.40.62:34783)
0928 15:01:58.474652 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:58.474658 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:58.475107 (+ 449us) server_negotiation.cc:407] Connection header received
0928 15:01:58.475277 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:58.475282 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:58.475381 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:58.475511 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:58.477177 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:58.478440 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:58.480350 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:58.481227 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:58.481457 (+ 230us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:58.484237 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:58.484244 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:58.484249 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:58.484294 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:58.488568 (+ 4274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:58.489871 (+ 1303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:58.489879 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:58.489885 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:58.490082 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:58.490724 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:58.490732 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:58.490738 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:58.491231 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:58.491982 (+ 751us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:58.492654 (+ 672us) server_negotiation.cc:299] Negotiation successful
0928 15:01:58.492928 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 15:01:58.497591 21338 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ef55f4c58864134bdb83a565341d91f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ef55f4c58864134bdb83a565341d91f. Latest consensus state: current_term: 1 leader_uuid: "0ef55f4c58864134bdb83a565341d91f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ef55f4c58864134bdb83a565341d91f" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34783 } } }
I20260928 15:01:58.498095 21338 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ef55f4c58864134bdb83a565341d91f [sys.catalog]: This master's current role is: LEADER
I20260928 15:01:58.502274 21337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ef55f4c58864134bdb83a565341d91f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ef55f4c58864134bdb83a565341d91f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ef55f4c58864134bdb83a565341d91f" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34783 } } }
I20260928 15:01:58.502610 21350 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:01:58.502643 21337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ef55f4c58864134bdb83a565341d91f [sys.catalog]: This master's current role is: LEADER
I20260928 15:01:58.508049 21350 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:01:58.517627 21350 catalog_manager.cc:1347] Generated new cluster ID: 0323929b23974c83a823491e1b8e72e3
I20260928 15:01:58.517767 21350 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:01:58.534615 21350 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:01:58.536621 21350 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:01:58.543849 21350 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ef55f4c58864134bdb83a565341d91f: Generated new TSK 0
I20260928 15:01:58.544982 21350 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:01:58.556052 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 21259
I20260928 15:01:58.557461 21321 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:01:58.652411 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 21259
W20260928 15:01:58.670809 21244 connection.cc:535] client connection to 127.31.40.62:34783 recv error: Network error: failed to read from TLS socket (remote: 127.31.40.62:34783): Connection reset by peer (error 104)
2026-09-28T15:01:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2072 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607365362981-31904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:01:58 dist-test-slave-j968 krb5kdc[21359](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:01:58 dist-test-slave-j968 krb5kdc[21359](info): set up 2 sockets
Sep 28 15:01:58 dist-test-slave-j968 krb5kdc[21359](info): commencing operation
krb5kdc: starting...
W20260928 15:01:59.782119 31904 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Sep 28 15:01:59 dist-test-slave-j968 krb5kdc[21359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:01:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:01:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab.
Entry for principal kudu/127.31.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab.
I20260928 15:02:00.059476 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607365362981-31904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607365362981-31904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607365362981-31904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:44869
--webserver_interface=127.31.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:45469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.40.62:44869
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:00.410535 21372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:00.410844 21372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:00.411022 21372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:00.418439 21372 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:02:00.418560 21372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:00.418672 21372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:00.418785 21372 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:02:00.418848 21372 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:02:00.429162 21372 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:45469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607365362981-31904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607365362981-31904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.40.62:44869
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:44869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607365362981-31904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:02:00.430555 21372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:00.431954 21372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:00.444912 21377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:00.444957 21380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:00.445466 21378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:00.446053 21372 server_base.cc:1034] running on GCE node
I20260928 15:02:00.446916 21372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:00.448657 21372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:00.449870 21372 hybrid_clock.cc:648] HybridClock initialized: now 1790607720449821 us; error 42 us; skew 500 ppm
I20260928 15:02:00.450341 21372 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:02:00 dist-test-slave-j968 krb5kdc[21359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607720, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:00.454203 21372 init.cc:377] Logged in from keytab as kudu/127.31.40.62@KRBTEST.COM (short username kudu)
I20260928 15:02:00.456228 21372 webserver.cc:460] Webserver started at http://127.31.40.62:40719/ using document root <none> and password file <none>
I20260928 15:02:00.457034 21372 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:00.457171 21372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:00.457528 21372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:00.461009 21372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607365362981-31904-0/minicluster-data/master-0/data/instance:
uuid: "eeedcea20158455b8645c2e13e3cb9bd"
format_stamp: "Formatted at 2026-09-28 15:02:00 on dist-test-slave-j968"
server_key: "3524a8bb90d198228781d6f47a5c1d68"
server_key_iv: "03798a67033799ae811edd5b13791626"
server_key_version: "encryptionkey@0"
I20260928 15:02:00.461928 21372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607365362981-31904-0/minicluster-data/master-0/wal/instance:
uuid: "eeedcea20158455b8645c2e13e3cb9bd"
format_stamp: "Formatted at 2026-09-28 15:02:00 on dist-test-slave-j968"
server_key: "3524a8bb90d198228781d6f47a5c1d68"
server_key_iv: "03798a67033799ae811edd5b13791626"
server_key_version: "encryptionkey@0"
I20260928 15:02:00.468235 21372 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260928 15:02:00.472081 21387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:00.473538 21372 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:02:00.473747 21372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607365362981-31904-0/minicluster-data/master-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607365362981-31904-0/minicluster-data/master-0/wal
uuid: "eeedcea20158455b8645c2e13e3cb9bd"
format_stamp: "Formatted at 2026-09-28 15:02:00 on dist-test-slave-j968"
server_key: "3524a8bb90d198228781d6f47a5c1d68"
server_key_iv: "03798a67033799ae811edd5b13791626"
server_key_version: "encryptionkey@0"
I20260928 15:02:00.473989 21372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607365362981-31904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607365362981-31904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607365362981-31904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:00.491297 21372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:00.496340 21372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:00.496670 21372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:00.516587 21372 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.62:44869
I20260928 15:02:00.516647 21438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.62:44869 every 8 connection(s)
I20260928 15:02:00.518102 21372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
I20260928 15:02:00.522161 21439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:00.527046 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 21372
I20260928 15:02:00.527235 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607365362981-31904-0/minicluster-data/master-0/wal/instance
I20260928 15:02:00.527789 31904 external_mini_cluster.cc:1428] Setting key 1f0e8291bafbb208adabfcde50763742
I20260928 15:02:00.534565 21439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eeedcea20158455b8645c2e13e3cb9bd: Bootstrap starting.
I20260928 15:02:00.537962 21439 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eeedcea20158455b8645c2e13e3cb9bd: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:00.539108 21439 log.cc:824] T 00000000000000000000000000000000 P eeedcea20158455b8645c2e13e3cb9bd: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:00.542652 21439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eeedcea20158455b8645c2e13e3cb9bd: No bootstrap required, opened a new log
I20260928 15:02:00.542675 21442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:00.529790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34061 (local address 127.31.40.62:44869)
0928 15:02:00.530699 (+ 909us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:00.530712 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:00.530751 (+ 39us) server_negotiation.cc:407] Connection header received
0928 15:02:00.533135 (+ 2384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:00.533163 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:00.533618 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:00.534043 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:00.534633 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:00.536112 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:00.537462 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:00.538254 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:00.538477 (+ 223us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:00.540028 (+ 1551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:00.540043 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:00.540064 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:02:00.540850 (+ 786us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:00.540933 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:02:00.540960 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:00.541141 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:00.541457 (+ 316us) server_negotiation.cc:299] Negotiation successful
0928 15:02:00.541638 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":366,"threads_started":1}
I20260928 15:02:00.547025 31904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:02:00.551416 21439 raft_consensus.cc:348] T 00000000000000000000000000000000 P eeedcea20158455b8645c2e13e3cb9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeedcea20158455b8645c2e13e3cb9bd" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 44869 } }
I20260928 15:02:00.552117 21439 raft_consensus.cc:374] T 00000000000000000000000000000000 P eeedcea20158455b8645c2e13e3cb9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:00.552256 21439 raft_consensus.cc:729] T 00000000000000000000000000000000 P eeedcea20158455b8645c2e13e3cb9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeedcea20158455b8645c2e13e3cb9bd, State: Initialized, Role: FOLLOWER
I20260928 15:02:00.552953 21439 consensus_queue.cc:260] T 00000000000000000000000000000000 P eeedcea20158455b8645c2e13e3cb9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeedcea20158455b8645c2e13e3cb9bd" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 44869 } }
I20260928 15:02:00.553222 21439 raft_consensus.cc:388] T 00000000000000000000000000000000 P eeedcea20158455b8645c2e13e3cb9bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:02:00.553395 21439 raft_consensus.cc:482] T 00000000000000000000000000000000 P eeedcea20158455b8645c2e13e3cb9bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:02:00.553583 21439 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eeedcea20158455b8645c2e13e3cb9bd [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:00.556115 21439 raft_consensus.cc:504] T 00000000000000000000000000000000 P eeedcea20158455b8645c2e13e3cb9bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeedcea20158455b8645c2e13e3cb9bd" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 44869 } }
I20260928 15:02:00.556700 21439 leader_election.cc:302] T 00000000000000000000000000000000 P eeedcea20158455b8645c2e13e3cb9bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eeedcea20158455b8645c2e13e3cb9bd; no voters:
I20260928 15:02:00.557437 21439 leader_election.cc:288] T 00000000000000000000000000000000 P eeedcea20158455b8645c2e13e3cb9bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:02:00.557538 21449 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eeedcea20158455b8645c2e13e3cb9bd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:00.558298 21449 raft_consensus.cc:686] T 00000000000000000000000000000000 P eeedcea20158455b8645c2e13e3cb9bd [term 1 LEADER]: Becoming Leader. State: Replica: eeedcea20158455b8645c2e13e3cb9bd, State: Running, Role: LEADER
I20260928 15:02:00.559103 21442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:00.551584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55639 (local address 127.31.40.62:44869)
0928 15:02:00.551756 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:00.551760 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:00.552645 (+ 885us) server_negotiation.cc:407] Connection header received
0928 15:02:00.552878 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:00.552882 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:00.552953 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:00.553060 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:00.553654 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:00.554478 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:00.555815 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:00.556432 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:00.556572 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:00.557967 (+ 1395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:00.557972 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:00.557979 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:02:00.557999 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:00.558062 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:02:00.558085 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:00.558226 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:00.558720 (+ 494us) server_negotiation.cc:299] Negotiation successful
0928 15:02:00.558774 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 15:02:00.559533 21449 consensus_queue.cc:237] T 00000000000000000000000000000000 P eeedcea20158455b8645c2e13e3cb9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeedcea20158455b8645c2e13e3cb9bd" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 44869 } }
I20260928 15:02:00.561390 21439 sys_catalog.cc:564] T 00000000000000000000000000000000 P eeedcea20158455b8645c2e13e3cb9bd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:02:00.562585 21450 sys_catalog.cc:455] T 00000000000000000000000000000000 P eeedcea20158455b8645c2e13e3cb9bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader eeedcea20158455b8645c2e13e3cb9bd. Latest consensus state: current_term: 1 leader_uuid: "eeedcea20158455b8645c2e13e3cb9bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeedcea20158455b8645c2e13e3cb9bd" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 44869 } } }
I20260928 15:02:00.562697 21451 sys_catalog.cc:455] T 00000000000000000000000000000000 P eeedcea20158455b8645c2e13e3cb9bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eeedcea20158455b8645c2e13e3cb9bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeedcea20158455b8645c2e13e3cb9bd" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 44869 } } }
I20260928 15:02:00.563051 21450 sys_catalog.cc:458] T 00000000000000000000000000000000 P eeedcea20158455b8645c2e13e3cb9bd [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:00.563051 21451 sys_catalog.cc:458] T 00000000000000000000000000000000 P eeedcea20158455b8645c2e13e3cb9bd [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:00.564786 21453 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:02:00.570672 21453 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:02:00.581123 21453 catalog_manager.cc:1347] Generated new cluster ID: 2311d95eb5ec48e8b00270f11cc8bd99
I20260928 15:02:00.581271 21453 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:02:00.594828 21453 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:02:00.596338 21453 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:02:00.601514 21453 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eeedcea20158455b8645c2e13e3cb9bd: Generated new TSK 0
I20260928 15:02:00.602368 21453 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:02:00.610088 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 21372
W20260928 15:02:00.612273 21434 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34061
W20260928 15:02:00.613085 31904 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 21372: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:02:00.613252 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 21372
W20260928 15:02:00.631034 21357 connection.cc:535] client connection to 127.31.40.62:44869 recv error: Network error: failed to read from TLS socket (remote: 127.31.40.62:44869): Connection reset by peer (error 104)
2026-09-28T15:02:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1959 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607365362981-31904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:02:00 dist-test-slave-j968 krb5kdc[21470](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:02:00 dist-test-slave-j968 krb5kdc[21470](info): set up 2 sockets
Sep 28 15:02:00 dist-test-slave-j968 krb5kdc[21470](info): commencing operation
krb5kdc: starting...
W20260928 15:02:01.733879 31904 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Sep 28 15:02:01 dist-test-slave-j968 krb5kdc[21470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:02:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:02:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab.
Entry for principal kudu/127.31.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab.
I20260928 15:02:02.023789 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607365362981-31904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607365362981-31904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607365362981-31904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:42371
--webserver_interface=127.31.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:35741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.40.62:42371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:02.382596 21483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:02.383074 21483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:02.383265 21483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:02.391641 21483 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:02:02.391769 21483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:02.391824 21483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:02.391927 21483 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:02:02.391996 21483 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:02:02.402604 21483 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:35741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607365362981-31904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607365362981-31904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.40.62:42371
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:42371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607365362981-31904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:02:02.404075 21483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:02.405464 21483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:02.418435 21489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:02.418547 21491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:02.418566 21488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:02.419178 21483 server_base.cc:1034] running on GCE node
I20260928 15:02:02.420194 21483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:02.421866 21483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:02.423139 21483 hybrid_clock.cc:648] HybridClock initialized: now 1790607722423082 us; error 47 us; skew 500 ppm
I20260928 15:02:02.423574 21483 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:02:02 dist-test-slave-j968 krb5kdc[21470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607722, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:02.432874 21483 init.cc:377] Logged in from keytab as kudu/127.31.40.62@KRBTEST.COM (short username kudu)
I20260928 15:02:02.434792 21483 webserver.cc:460] Webserver started at http://127.31.40.62:34035/ using document root <none> and password file <none>
I20260928 15:02:02.435629 21483 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:02.435803 21483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:02.436190 21483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:02.439646 21483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607365362981-31904-0/minicluster-data/master-0/data/instance:
uuid: "b0217bb32cfd44059d77f25516ea23ae"
format_stamp: "Formatted at 2026-09-28 15:02:02 on dist-test-slave-j968"
server_key: "3ccf68a914176c4abb4b7fa4345555ed"
server_key_iv: "7bea50d2a488698f192b65837b923b38"
server_key_version: "encryptionkey@0"
I20260928 15:02:02.440670 21483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607365362981-31904-0/minicluster-data/master-0/wal/instance:
uuid: "b0217bb32cfd44059d77f25516ea23ae"
format_stamp: "Formatted at 2026-09-28 15:02:02 on dist-test-slave-j968"
server_key: "3ccf68a914176c4abb4b7fa4345555ed"
server_key_iv: "7bea50d2a488698f192b65837b923b38"
server_key_version: "encryptionkey@0"
I20260928 15:02:02.447096 21483 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260928 15:02:02.451663 21498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:02.453498 21483 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 15:02:02.453740 21483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607365362981-31904-0/minicluster-data/master-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607365362981-31904-0/minicluster-data/master-0/wal
uuid: "b0217bb32cfd44059d77f25516ea23ae"
format_stamp: "Formatted at 2026-09-28 15:02:02 on dist-test-slave-j968"
server_key: "3ccf68a914176c4abb4b7fa4345555ed"
server_key_iv: "7bea50d2a488698f192b65837b923b38"
server_key_version: "encryptionkey@0"
I20260928 15:02:02.453994 21483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607365362981-31904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607365362981-31904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607365362981-31904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:02.461246 21483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:02.466908 21483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:02.467320 21483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:02.488556 21483 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.62:42371
I20260928 15:02:02.488605 21549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.62:42371 every 8 connection(s)
I20260928 15:02:02.490200 21483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
I20260928 15:02:02.491719 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 21483
I20260928 15:02:02.491932 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607365362981-31904-0/minicluster-data/master-0/wal/instance
I20260928 15:02:02.492502 31904 external_mini_cluster.cc:1428] Setting key 16e542833e3d46609161558e1e7f7fc7
I20260928 15:02:02.495543 21550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:02.502355 21553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:02.494543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39177 (local address 127.31.40.62:42371)
0928 15:02:02.495456 (+ 913us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:02.495475 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:02.495541 (+ 66us) server_negotiation.cc:407] Connection header received
0928 15:02:02.497964 (+ 2423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:02.497995 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:02.498456 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:02.498986 (+ 530us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:02.500167 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:02.500181 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:02.500199 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:02:02.500219 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:02.500324 (+ 105us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:02:02.500363 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:02.500599 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:02.500736 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 15:02:02.500977 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":416,"threads_started":1}
I20260928 15:02:02.506738 31904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:02:02.510481 21550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0217bb32cfd44059d77f25516ea23ae: Bootstrap starting.
I20260928 15:02:02.515587 21550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0217bb32cfd44059d77f25516ea23ae: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:02.516804 21553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:02.513693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56159 (local address 127.31.40.62:42371)
0928 15:02:02.513939 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:02.513945 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:02.514366 (+ 421us) server_negotiation.cc:407] Connection header received
0928 15:02:02.514596 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:02.514601 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:02.514657 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:02.514751 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:02.515917 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:02.515924 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:02.515929 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:02:02.515955 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:02.516002 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:02:02.516019 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:02.516127 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:02.516349 (+ 222us) server_negotiation.cc:299] Negotiation successful
0928 15:02:02.516447 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 15:02:02.517172 21550 log.cc:824] T 00000000000000000000000000000000 P b0217bb32cfd44059d77f25516ea23ae: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:02.520488 21550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0217bb32cfd44059d77f25516ea23ae: No bootstrap required, opened a new log
I20260928 15:02:02.528095 21550 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0217bb32cfd44059d77f25516ea23ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0217bb32cfd44059d77f25516ea23ae" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 42371 } }
I20260928 15:02:02.528612 21550 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0217bb32cfd44059d77f25516ea23ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:02.528733 21550 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0217bb32cfd44059d77f25516ea23ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0217bb32cfd44059d77f25516ea23ae, State: Initialized, Role: FOLLOWER
I20260928 15:02:02.529352 21550 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0217bb32cfd44059d77f25516ea23ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0217bb32cfd44059d77f25516ea23ae" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 42371 } }
I20260928 15:02:02.529568 21550 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0217bb32cfd44059d77f25516ea23ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:02:02.529667 21550 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0217bb32cfd44059d77f25516ea23ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:02:02.529794 21550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0217bb32cfd44059d77f25516ea23ae [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:02.532064 21550 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0217bb32cfd44059d77f25516ea23ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0217bb32cfd44059d77f25516ea23ae" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 42371 } }
I20260928 15:02:02.532569 21550 leader_election.cc:302] T 00000000000000000000000000000000 P b0217bb32cfd44059d77f25516ea23ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0217bb32cfd44059d77f25516ea23ae; no voters:
I20260928 15:02:02.533200 21550 leader_election.cc:288] T 00000000000000000000000000000000 P b0217bb32cfd44059d77f25516ea23ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:02:02.533355 21560 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0217bb32cfd44059d77f25516ea23ae [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:02.534768 21560 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0217bb32cfd44059d77f25516ea23ae [term 1 LEADER]: Becoming Leader. State: Replica: b0217bb32cfd44059d77f25516ea23ae, State: Running, Role: LEADER
I20260928 15:02:02.535357 21560 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0217bb32cfd44059d77f25516ea23ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0217bb32cfd44059d77f25516ea23ae" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 42371 } }
I20260928 15:02:02.535754 21550 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0217bb32cfd44059d77f25516ea23ae [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:02:02.538854 21562 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0217bb32cfd44059d77f25516ea23ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0217bb32cfd44059d77f25516ea23ae. Latest consensus state: current_term: 1 leader_uuid: "b0217bb32cfd44059d77f25516ea23ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0217bb32cfd44059d77f25516ea23ae" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 42371 } } }
I20260928 15:02:02.539387 21562 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0217bb32cfd44059d77f25516ea23ae [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:02.538913 21561 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0217bb32cfd44059d77f25516ea23ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0217bb32cfd44059d77f25516ea23ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0217bb32cfd44059d77f25516ea23ae" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 42371 } } }
I20260928 15:02:02.540041 21561 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0217bb32cfd44059d77f25516ea23ae [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:02.546245 21566 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:02:02.552764 21566 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:02:02.565590 21566 catalog_manager.cc:1347] Generated new cluster ID: ce0b9ce8752242ca957f5e6adc3024c5
I20260928 15:02:02.565747 21566 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:02:02.608134 21566 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:02:02.608711 21566 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:02:02.615525 21566 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0217bb32cfd44059d77f25516ea23ae: Generated new TSK 0
I20260928 15:02:02.616482 21566 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:02:02.628494 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 21483
W20260928 15:02:02.629899 21545 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:39177
W20260928 15:02:02.630681 31904 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 21483: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:02:02.630847 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 21483
2026-09-28T15:02:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2019 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607365362981-31904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:02:02 dist-test-slave-j968 krb5kdc[21581](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:02:02 dist-test-slave-j968 krb5kdc[21581](info): set up 2 sockets
Sep 28 15:02:02 dist-test-slave-j968 krb5kdc[21581](info): commencing operation
krb5kdc: starting...
W20260928 15:02:03.750203 31904 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Sep 28 15:02:03 dist-test-slave-j968 krb5kdc[21581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:02:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:02:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab.
Entry for principal kudu/127.31.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab.
I20260928 15:02:04.035727 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607365362981-31904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607365362981-31904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607365362981-31904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:39791
--webserver_interface=127.31.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:46629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.40.62:39791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:04.392184 21594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:04.392642 21594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:04.392812 21594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:04.400355 21594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:02:04.400491 21594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:04.400537 21594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:04.400679 21594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:02:04.400746 21594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:02:04.412376 21594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:46629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607365362981-31904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607365362981-31904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.40.62:39791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:39791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607365362981-31904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:02:04.413976 21594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:04.415509 21594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:04.429560 21599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:04.429683 21602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:04.429657 21600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:04.431139 21594 server_base.cc:1034] running on GCE node
I20260928 15:02:04.432246 21594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:04.434144 21594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:04.435441 21594 hybrid_clock.cc:648] HybridClock initialized: now 1790607724435392 us; error 41 us; skew 500 ppm
I20260928 15:02:04.435969 21594 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:02:04 dist-test-slave-j968 krb5kdc[21581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607724, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:04.439574 21594 init.cc:377] Logged in from keytab as kudu/127.31.40.62@KRBTEST.COM (short username kudu)
I20260928 15:02:04.441629 21594 webserver.cc:460] Webserver started at http://127.31.40.62:42767/ using document root <none> and password file <none>
I20260928 15:02:04.442476 21594 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:04.442646 21594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:04.442991 21594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:04.446516 21594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607365362981-31904-0/minicluster-data/master-0/data/instance:
uuid: "97346167eaf04cbb8146a0d512dc15b0"
format_stamp: "Formatted at 2026-09-28 15:02:04 on dist-test-slave-j968"
server_key: "f9402f5145ba140a06e58c7ae6a2c6e7"
server_key_iv: "80c16a20f8f17f23de354e2242e64252"
server_key_version: "encryptionkey@0"
I20260928 15:02:04.447422 21594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607365362981-31904-0/minicluster-data/master-0/wal/instance:
uuid: "97346167eaf04cbb8146a0d512dc15b0"
format_stamp: "Formatted at 2026-09-28 15:02:04 on dist-test-slave-j968"
server_key: "f9402f5145ba140a06e58c7ae6a2c6e7"
server_key_iv: "80c16a20f8f17f23de354e2242e64252"
server_key_version: "encryptionkey@0"
I20260928 15:02:04.454542 21594 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.002s
I20260928 15:02:04.459233 21609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:04.461243 21594 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 15:02:04.461663 21594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607365362981-31904-0/minicluster-data/master-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607365362981-31904-0/minicluster-data/master-0/wal
uuid: "97346167eaf04cbb8146a0d512dc15b0"
format_stamp: "Formatted at 2026-09-28 15:02:04 on dist-test-slave-j968"
server_key: "f9402f5145ba140a06e58c7ae6a2c6e7"
server_key_iv: "80c16a20f8f17f23de354e2242e64252"
server_key_version: "encryptionkey@0"
I20260928 15:02:04.461967 21594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607365362981-31904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607365362981-31904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607365362981-31904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:04.498948 21594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:04.504184 21594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:04.504511 21594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:04.525127 21594 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.62:39791
I20260928 15:02:04.525295 21660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.62:39791 every 8 connection(s)
I20260928 15:02:04.526652 21594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
I20260928 15:02:04.530766 21661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:04.535676 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 21594
I20260928 15:02:04.536046 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607365362981-31904-0/minicluster-data/master-0/wal/instance
I20260928 15:02:04.536553 31904 external_mini_cluster.cc:1428] Setting key d36a057b6f903e202ccfa650cc88eccd
I20260928 15:02:04.543004 21661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97346167eaf04cbb8146a0d512dc15b0: Bootstrap starting.
I20260928 15:02:04.547096 21661 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97346167eaf04cbb8146a0d512dc15b0: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:04.548738 21661 log.cc:824] T 00000000000000000000000000000000 P 97346167eaf04cbb8146a0d512dc15b0: Log is configured to *not* fsync() on all Append() calls
Sep 28 15:02:04 dist-test-slave-j968 krb5kdc[21581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 15:02:04.553558 21661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97346167eaf04cbb8146a0d512dc15b0: No bootstrap required, opened a new log
I20260928 15:02:04.563452 21661 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97346167eaf04cbb8146a0d512dc15b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97346167eaf04cbb8146a0d512dc15b0" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 39791 } }
I20260928 15:02:04.564138 21661 raft_consensus.cc:374] T 00000000000000000000000000000000 P 97346167eaf04cbb8146a0d512dc15b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:04.564275 21661 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97346167eaf04cbb8146a0d512dc15b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97346167eaf04cbb8146a0d512dc15b0, State: Initialized, Role: FOLLOWER
I20260928 15:02:04.565065 21661 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97346167eaf04cbb8146a0d512dc15b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97346167eaf04cbb8146a0d512dc15b0" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 39791 } }
I20260928 15:02:04.565275 21664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:04.538716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55637 (local address 127.31.40.62:39791)
0928 15:02:04.539650 (+ 934us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:04.539667 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:04.539715 (+ 48us) server_negotiation.cc:407] Connection header received
0928 15:02:04.541631 (+ 1916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:04.541659 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:04.542158 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:04.542539 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:04.544542 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:04.546018 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:04.548030 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:04.548808 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:04.549006 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:04.555260 (+ 6254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:04.555439 (+ 179us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:04.555462 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:04.555529 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:04.559629 (+ 4100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:04.560611 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:04.560620 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:04.560630 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:04.560850 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:04.561389 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:04.561395 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:04.561400 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:04.562176 (+ 776us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:04.562565 (+ 389us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:04.563090 (+ 525us) server_negotiation.cc:299] Negotiation successful
0928 15:02:04.563674 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":694,"thread_start_us":443,"threads_started":1}
I20260928 15:02:04.565407 21661 raft_consensus.cc:388] T 00000000000000000000000000000000 P 97346167eaf04cbb8146a0d512dc15b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:02:04.565575 21661 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97346167eaf04cbb8146a0d512dc15b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:02:04.565788 21661 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97346167eaf04cbb8146a0d512dc15b0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:04.569319 21661 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97346167eaf04cbb8146a0d512dc15b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97346167eaf04cbb8146a0d512dc15b0" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 39791 } }
I20260928 15:02:04.569998 21661 leader_election.cc:302] T 00000000000000000000000000000000 P 97346167eaf04cbb8146a0d512dc15b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97346167eaf04cbb8146a0d512dc15b0; no voters:
I20260928 15:02:04.570533 31904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:02:04.570765 21661 leader_election.cc:288] T 00000000000000000000000000000000 P 97346167eaf04cbb8146a0d512dc15b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:02:04.570883 21666 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97346167eaf04cbb8146a0d512dc15b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:04.573962 21661 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97346167eaf04cbb8146a0d512dc15b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:02:04.577059 21666 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97346167eaf04cbb8146a0d512dc15b0 [term 1 LEADER]: Becoming Leader. State: Replica: 97346167eaf04cbb8146a0d512dc15b0, State: Running, Role: LEADER
I20260928 15:02:04.578161 21666 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97346167eaf04cbb8146a0d512dc15b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97346167eaf04cbb8146a0d512dc15b0" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 39791 } }
I20260928 15:02:04.582636 21667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97346167eaf04cbb8146a0d512dc15b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "97346167eaf04cbb8146a0d512dc15b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97346167eaf04cbb8146a0d512dc15b0" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 39791 } } }
I20260928 15:02:04.583233 21667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97346167eaf04cbb8146a0d512dc15b0 [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:04.584102 21668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97346167eaf04cbb8146a0d512dc15b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97346167eaf04cbb8146a0d512dc15b0. Latest consensus state: current_term: 1 leader_uuid: "97346167eaf04cbb8146a0d512dc15b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97346167eaf04cbb8146a0d512dc15b0" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 39791 } } }
I20260928 15:02:04.584564 21668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97346167eaf04cbb8146a0d512dc15b0 [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:04.604657 21664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:04.579586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52101 (local address 127.31.40.62:39791)
0928 15:02:04.580156 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:04.580164 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:04.580191 (+ 27us) server_negotiation.cc:407] Connection header received
0928 15:02:04.580452 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:04.580460 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:04.580567 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:04.580690 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:04.583069 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:04.584433 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:04.587186 (+ 2753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:04.588255 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:04.588451 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:04.593191 (+ 4740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:04.593203 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:04.593211 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:04.593283 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:04.598649 (+ 5366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:04.600132 (+ 1483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:04.600145 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:04.600152 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:04.600393 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:04.601038 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:04.601047 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:04.601053 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:04.601610 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:04.601967 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:04.603698 (+ 1731us) server_negotiation.cc:299] Negotiation successful
0928 15:02:04.604085 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440}
I20260928 15:02:04.607311 21682 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:02:04.612389 21682 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:02:04.622923 21682 catalog_manager.cc:1347] Generated new cluster ID: dc0e2ca4a07b4cc1915a1f041add2500
I20260928 15:02:04.623093 21682 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:02:04.643595 21682 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:02:04.645262 21682 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:02:04.657341 21682 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 97346167eaf04cbb8146a0d512dc15b0: Generated new TSK 0
I20260928 15:02:04.658478 21682 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:02:04.678537 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 21594
I20260928 15:02:04.680325 21656 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:02:04.777990 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 21594
2026-09-28T15:02:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2146 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607365362981-31904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:02:04 dist-test-slave-j968 krb5kdc[21694](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:02:04 dist-test-slave-j968 krb5kdc[21694](info): set up 2 sockets
Sep 28 15:02:04 dist-test-slave-j968 krb5kdc[21694](info): commencing operation
krb5kdc: starting...
W20260928 15:02:05.901419 31904 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Sep 28 15:02:06 dist-test-slave-j968 krb5kdc[21694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:02:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:02:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab.
Entry for principal kudu/127.31.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab.
I20260928 15:02:06.200109 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607365362981-31904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607365362981-31904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607365362981-31904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:33411
--webserver_interface=127.31.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:44567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.40.62:33411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:06.548830 21707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:06.549146 21707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:06.549314 21707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:06.557005 21707 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:02:06.557129 21707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:06.557251 21707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:06.557331 21707 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:02:06.557369 21707 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:02:06.567648 21707 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:44567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607365362981-31904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607365362981-31904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.40.62:33411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:33411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607365362981-31904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:02:06.569145 21707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:06.570578 21707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:06.583138 21715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:06.583211 21712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:06.583091 21713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:06.584122 21707 server_base.cc:1034] running on GCE node
I20260928 15:02:06.585017 21707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:06.586544 21707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:06.587785 21707 hybrid_clock.cc:648] HybridClock initialized: now 1790607726587730 us; error 42 us; skew 500 ppm
I20260928 15:02:06.588195 21707 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:02:06 dist-test-slave-j968 krb5kdc[21694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607726, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:06.591584 21707 init.cc:377] Logged in from keytab as kudu/127.31.40.62@KRBTEST.COM (short username kudu)
I20260928 15:02:06.593349 21707 webserver.cc:460] Webserver started at http://127.31.40.62:40327/ using document root <none> and password file <none>
I20260928 15:02:06.594079 21707 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:06.594214 21707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:06.594533 21707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:06.597744 21707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607365362981-31904-0/minicluster-data/master-0/data/instance:
uuid: "b7a3d7b950ff4fb49e060f0e2ff7a47f"
format_stamp: "Formatted at 2026-09-28 15:02:06 on dist-test-slave-j968"
server_key: "b94e022dc401af28d6d6a4318e727901"
server_key_iv: "b67fd2a052dd26f6a76737cfdc3f554e"
server_key_version: "encryptionkey@0"
I20260928 15:02:06.598531 21707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607365362981-31904-0/minicluster-data/master-0/wal/instance:
uuid: "b7a3d7b950ff4fb49e060f0e2ff7a47f"
format_stamp: "Formatted at 2026-09-28 15:02:06 on dist-test-slave-j968"
server_key: "b94e022dc401af28d6d6a4318e727901"
server_key_iv: "b67fd2a052dd26f6a76737cfdc3f554e"
server_key_version: "encryptionkey@0"
I20260928 15:02:06.604081 21707 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 15:02:06.608253 21722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:06.609807 21707 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:02:06.610023 21707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607365362981-31904-0/minicluster-data/master-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607365362981-31904-0/minicluster-data/master-0/wal
uuid: "b7a3d7b950ff4fb49e060f0e2ff7a47f"
format_stamp: "Formatted at 2026-09-28 15:02:06 on dist-test-slave-j968"
server_key: "b94e022dc401af28d6d6a4318e727901"
server_key_iv: "b67fd2a052dd26f6a76737cfdc3f554e"
server_key_version: "encryptionkey@0"
I20260928 15:02:06.610255 21707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607365362981-31904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607365362981-31904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607365362981-31904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:06.629093 21707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:06.633818 21707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:06.634124 21707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:06.653327 21707 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.62:33411
I20260928 15:02:06.653362 21773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.62:33411 every 8 connection(s)
I20260928 15:02:06.654763 21707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
I20260928 15:02:06.657364 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 21707
I20260928 15:02:06.657553 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607365362981-31904-0/minicluster-data/master-0/wal/instance
I20260928 15:02:06.658016 31904 external_mini_cluster.cc:1428] Setting key 93642807ee2b8502fcfc8e1ba458532b
I20260928 15:02:06.659179 21774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:06.673609 21777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:06.660005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36295 (local address 127.31.40.62:33411)
0928 15:02:06.660818 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:06.660836 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:06.660888 (+ 52us) server_negotiation.cc:407] Connection header received
0928 15:02:06.662760 (+ 1872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:06.662790 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:06.663339 (+ 549us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:06.663828 (+ 489us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:06.664400 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:06.666471 (+ 2071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:06.667827 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:06.668892 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:06.669076 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:06.670398 (+ 1322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:06.670408 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:06.670427 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:02:06.671237 (+ 810us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:06.671341 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:02:06.671374 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:06.671513 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:06.671749 (+ 236us) server_negotiation.cc:299] Negotiation successful
0928 15:02:06.672077 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":345,"threads_started":1}
I20260928 15:02:06.674573 21774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7a3d7b950ff4fb49e060f0e2ff7a47f: Bootstrap starting.
I20260928 15:02:06.677577 31904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:02:06.678100 21774 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7a3d7b950ff4fb49e060f0e2ff7a47f: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:06.679143 21774 log.cc:824] T 00000000000000000000000000000000 P b7a3d7b950ff4fb49e060f0e2ff7a47f: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:06.682385 21774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7a3d7b950ff4fb49e060f0e2ff7a47f: No bootstrap required, opened a new log
I20260928 15:02:06.689393 21777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:06.681999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42257 (local address 127.31.40.62:33411)
0928 15:02:06.682252 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:06.682261 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:06.682638 (+ 377us) server_negotiation.cc:407] Connection header received
0928 15:02:06.682857 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:06.682861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:06.682923 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:06.683011 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:06.683725 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:06.684722 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:06.686668 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:06.687250 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:06.687368 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:06.688550 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:06.688555 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:06.688558 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:02:06.688584 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:06.688628 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:02:06.688644 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:06.688726 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:06.688912 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 15:02:06.689000 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 15:02:06.692204 21774 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7a3d7b950ff4fb49e060f0e2ff7a47f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7a3d7b950ff4fb49e060f0e2ff7a47f" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 33411 } }
I20260928 15:02:06.692639 21774 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7a3d7b950ff4fb49e060f0e2ff7a47f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:06.692751 21774 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7a3d7b950ff4fb49e060f0e2ff7a47f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7a3d7b950ff4fb49e060f0e2ff7a47f, State: Initialized, Role: FOLLOWER
I20260928 15:02:06.693361 21774 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7a3d7b950ff4fb49e060f0e2ff7a47f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7a3d7b950ff4fb49e060f0e2ff7a47f" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 33411 } }
I20260928 15:02:06.693612 21774 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7a3d7b950ff4fb49e060f0e2ff7a47f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:02:06.693760 21774 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7a3d7b950ff4fb49e060f0e2ff7a47f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:02:06.693950 21774 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7a3d7b950ff4fb49e060f0e2ff7a47f [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:06.696390 21774 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7a3d7b950ff4fb49e060f0e2ff7a47f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7a3d7b950ff4fb49e060f0e2ff7a47f" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 33411 } }
I20260928 15:02:06.696919 21774 leader_election.cc:302] T 00000000000000000000000000000000 P b7a3d7b950ff4fb49e060f0e2ff7a47f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7a3d7b950ff4fb49e060f0e2ff7a47f; no voters:
I20260928 15:02:06.697531 21774 leader_election.cc:288] T 00000000000000000000000000000000 P b7a3d7b950ff4fb49e060f0e2ff7a47f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:02:06.697754 21784 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7a3d7b950ff4fb49e060f0e2ff7a47f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:06.698540 21784 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7a3d7b950ff4fb49e060f0e2ff7a47f [term 1 LEADER]: Becoming Leader. State: Replica: b7a3d7b950ff4fb49e060f0e2ff7a47f, State: Running, Role: LEADER
I20260928 15:02:06.699160 21784 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7a3d7b950ff4fb49e060f0e2ff7a47f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7a3d7b950ff4fb49e060f0e2ff7a47f" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 33411 } }
I20260928 15:02:06.700024 21774 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7a3d7b950ff4fb49e060f0e2ff7a47f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:02:06.702286 21786 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7a3d7b950ff4fb49e060f0e2ff7a47f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7a3d7b950ff4fb49e060f0e2ff7a47f. Latest consensus state: current_term: 1 leader_uuid: "b7a3d7b950ff4fb49e060f0e2ff7a47f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7a3d7b950ff4fb49e060f0e2ff7a47f" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 33411 } } }
I20260928 15:02:06.703127 21786 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7a3d7b950ff4fb49e060f0e2ff7a47f [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:06.702690 21785 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7a3d7b950ff4fb49e060f0e2ff7a47f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7a3d7b950ff4fb49e060f0e2ff7a47f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7a3d7b950ff4fb49e060f0e2ff7a47f" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 33411 } } }
I20260928 15:02:06.704099 21785 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7a3d7b950ff4fb49e060f0e2ff7a47f [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:06.704999 21791 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:02:06.712364 21791 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:02:06.726100 21791 catalog_manager.cc:1347] Generated new cluster ID: e58c3eb543ab40fa89cbab47625a2345
I20260928 15:02:06.726240 21791 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:02:06.738754 21791 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:02:06.740902 21791 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:02:06.759658 21791 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7a3d7b950ff4fb49e060f0e2ff7a47f: Generated new TSK 0
I20260928 15:02:06.761476 21791 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:02:06.780444 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 21707
W20260928 15:02:06.781658 21769 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36295
W20260928 15:02:06.782490 31904 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 21707: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:02:06.782656 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 21707
2026-09-28T15:02:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2004 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607365362981-31904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:02:06 dist-test-slave-j968 krb5kdc[21805](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:02:06 dist-test-slave-j968 krb5kdc[21805](info): set up 2 sockets
Sep 28 15:02:06 dist-test-slave-j968 krb5kdc[21805](info): commencing operation
krb5kdc: starting...
W20260928 15:02:07.901353 31904 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Sep 28 15:02:08 dist-test-slave-j968 krb5kdc[21805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:02:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:02:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab.
Entry for principal kudu/127.31.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab.
I20260928 15:02:08.183735 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607365362981-31904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607365362981-31904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607365362981-31904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:36911
--webserver_interface=127.31.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:36875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.40.62:36911
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:08.525451 21818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:08.525825 21818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:08.525976 21818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:08.533483 21818 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:02:08.533654 21818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:08.533757 21818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:08.533934 21818 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:02:08.534011 21818 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:02:08.544360 21818 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:36875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607365362981-31904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607365362981-31904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.40.62:36911
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:36911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607365362981-31904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:02:08.545766 21818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:08.547173 21818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:08.560359 21823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:08.560323 21826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:08.560350 21824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:08.562310 21818 server_base.cc:1034] running on GCE node
I20260928 15:02:08.563446 21818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:08.565217 21818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:08.566497 21818 hybrid_clock.cc:648] HybridClock initialized: now 1790607728566424 us; error 68 us; skew 500 ppm
I20260928 15:02:08.567004 21818 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:02:08 dist-test-slave-j968 krb5kdc[21805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607728, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:08.580188 21818 init.cc:377] Logged in from keytab as kudu/127.31.40.62@KRBTEST.COM (short username kudu)
I20260928 15:02:08.582098 21818 webserver.cc:460] Webserver started at http://127.31.40.62:38181/ using document root <none> and password file <none>
I20260928 15:02:08.582897 21818 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:08.583068 21818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:08.583411 21818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:08.586603 21818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607365362981-31904-0/minicluster-data/master-0/data/instance:
uuid: "89f85c9cdd924721a0bc660b2589460b"
format_stamp: "Formatted at 2026-09-28 15:02:08 on dist-test-slave-j968"
server_key: "1ccd1079df8f6452593d1c0abe87cfef"
server_key_iv: "6cac0278c505c28914b59f68c0711391"
server_key_version: "encryptionkey@0"
I20260928 15:02:08.587500 21818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607365362981-31904-0/minicluster-data/master-0/wal/instance:
uuid: "89f85c9cdd924721a0bc660b2589460b"
format_stamp: "Formatted at 2026-09-28 15:02:08 on dist-test-slave-j968"
server_key: "1ccd1079df8f6452593d1c0abe87cfef"
server_key_iv: "6cac0278c505c28914b59f68c0711391"
server_key_version: "encryptionkey@0"
I20260928 15:02:08.593034 21818 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 15:02:08.597036 21833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:08.598423 21818 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 15:02:08.598631 21818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607365362981-31904-0/minicluster-data/master-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607365362981-31904-0/minicluster-data/master-0/wal
uuid: "89f85c9cdd924721a0bc660b2589460b"
format_stamp: "Formatted at 2026-09-28 15:02:08 on dist-test-slave-j968"
server_key: "1ccd1079df8f6452593d1c0abe87cfef"
server_key_iv: "6cac0278c505c28914b59f68c0711391"
server_key_version: "encryptionkey@0"
I20260928 15:02:08.598934 21818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607365362981-31904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607365362981-31904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607365362981-31904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:08.605490 21818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:08.610173 21818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:08.610520 21818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:08.631182 21818 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.62:36911
I20260928 15:02:08.631232 21884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.62:36911 every 8 connection(s)
I20260928 15:02:08.632925 21818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
I20260928 15:02:08.636758 21885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:08.640797 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 21818
I20260928 15:02:08.641115 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607365362981-31904-0/minicluster-data/master-0/wal/instance
I20260928 15:02:08.641692 31904 external_mini_cluster.cc:1428] Setting key 36e73a53f5a54e787317362094ade5c5
I20260928 15:02:08.648335 21885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89f85c9cdd924721a0bc660b2589460b: Bootstrap starting.
I20260928 15:02:08.650828 21888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:08.643769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43637 (local address 127.31.40.62:36911)
0928 15:02:08.644722 (+ 953us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:08.644736 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:08.644788 (+ 52us) server_negotiation.cc:407] Connection header received
0928 15:02:08.646963 (+ 2175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:08.646989 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:08.647390 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:08.647775 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:08.649019 (+ 1244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:08.649026 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:08.649041 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:02:08.649056 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:08.649120 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:02:08.649141 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:08.649337 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:08.649458 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 15:02:08.649709 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":710,"thread_start_us":408,"threads_started":1}
I20260928 15:02:08.652117 21885 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89f85c9cdd924721a0bc660b2589460b: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:08.653170 21885 log.cc:824] T 00000000000000000000000000000000 P 89f85c9cdd924721a0bc660b2589460b: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:08.655301 31904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:02:08.656418 21885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89f85c9cdd924721a0bc660b2589460b: No bootstrap required, opened a new log
I20260928 15:02:08.664100 21888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:08.660138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47329 (local address 127.31.40.62:36911)
0928 15:02:08.660337 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:08.660344 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:08.661853 (+ 1509us) server_negotiation.cc:407] Connection header received
0928 15:02:08.662032 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:08.662036 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:08.662094 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:08.662187 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:08.663219 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:08.663225 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:08.663229 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:02:08.663253 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:08.663297 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:02:08.663314 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:08.663429 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:08.663598 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 15:02:08.663652 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 15:02:08.665553 21885 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89f85c9cdd924721a0bc660b2589460b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89f85c9cdd924721a0bc660b2589460b" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 36911 } }
I20260928 15:02:08.665938 21885 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89f85c9cdd924721a0bc660b2589460b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:08.666042 21885 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89f85c9cdd924721a0bc660b2589460b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f85c9cdd924721a0bc660b2589460b, State: Initialized, Role: FOLLOWER
I20260928 15:02:08.666653 21885 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89f85c9cdd924721a0bc660b2589460b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89f85c9cdd924721a0bc660b2589460b" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 36911 } }
I20260928 15:02:08.666879 21885 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89f85c9cdd924721a0bc660b2589460b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:02:08.667054 21885 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89f85c9cdd924721a0bc660b2589460b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:02:08.667244 21885 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89f85c9cdd924721a0bc660b2589460b [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:08.669598 21885 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89f85c9cdd924721a0bc660b2589460b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89f85c9cdd924721a0bc660b2589460b" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 36911 } }
I20260928 15:02:08.670130 21885 leader_election.cc:302] T 00000000000000000000000000000000 P 89f85c9cdd924721a0bc660b2589460b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89f85c9cdd924721a0bc660b2589460b; no voters:
I20260928 15:02:08.670944 21885 leader_election.cc:288] T 00000000000000000000000000000000 P 89f85c9cdd924721a0bc660b2589460b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:02:08.671116 21895 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89f85c9cdd924721a0bc660b2589460b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:08.672056 21895 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89f85c9cdd924721a0bc660b2589460b [term 1 LEADER]: Becoming Leader. State: Replica: 89f85c9cdd924721a0bc660b2589460b, State: Running, Role: LEADER
I20260928 15:02:08.672813 21895 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89f85c9cdd924721a0bc660b2589460b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89f85c9cdd924721a0bc660b2589460b" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 36911 } }
I20260928 15:02:08.673237 21885 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89f85c9cdd924721a0bc660b2589460b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:02:08.675884 21897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89f85c9cdd924721a0bc660b2589460b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89f85c9cdd924721a0bc660b2589460b. Latest consensus state: current_term: 1 leader_uuid: "89f85c9cdd924721a0bc660b2589460b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89f85c9cdd924721a0bc660b2589460b" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 36911 } } }
I20260928 15:02:08.676347 21897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89f85c9cdd924721a0bc660b2589460b [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:08.676874 21896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89f85c9cdd924721a0bc660b2589460b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89f85c9cdd924721a0bc660b2589460b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89f85c9cdd924721a0bc660b2589460b" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 36911 } } }
I20260928 15:02:08.677178 21896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89f85c9cdd924721a0bc660b2589460b [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:08.678494 21902 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:02:08.684897 21902 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:02:08.694705 21902 catalog_manager.cc:1347] Generated new cluster ID: caaf294cc940480d93266a3483672c8a
I20260928 15:02:08.694847 21902 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:02:08.712672 21902 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:02:08.713161 21902 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:02:08.721575 21902 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89f85c9cdd924721a0bc660b2589460b: Generated new TSK 0
I20260928 15:02:08.722561 21902 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:02:08.729426 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 21818
W20260928 15:02:08.730816 21880 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43637
W20260928 15:02:08.731669 31904 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 21818: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:02:08.731938 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 21818
2026-09-28T15:02:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1946 ms)
[----------] 18 tests from AuthTokenIssuingTest (36412 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:02:08 dist-test-slave-j968 krb5kdc[21916](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:02:08 dist-test-slave-j968 krb5kdc[21916](info): set up 2 sockets
Sep 28 15:02:08 dist-test-slave-j968 krb5kdc[21916](info): commencing operation
krb5kdc: starting...
W20260928 15:02:09.869699 31904 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.010s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Sep 28 15:02:10 dist-test-slave-j968 krb5kdc[21916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:02:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:02:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab.
Entry for principal kudu/127.31.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab.
I20260928 15:02:10.171494 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:34185
--webserver_interface=127.31.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:36313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.40.62:34185,127.31.40.61:36039,127.31.40.60:39205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:10.509541 21929 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 15:02:10.509929 21929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:10.510027 21929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:10.510182 21929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:10.517560 21929 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:02:10.517654 21929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:10.517763 21929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:10.517863 21929 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:02:10.517972 21929 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:02:10.527998 21929 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:36313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.40.62:34185,127.31.40.61:36039,127.31.40.60:39205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:34185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:02:10.529402 21929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:10.530794 21929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:10.542856 21935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:10.542973 21934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:10.542981 21937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:10.544158 21929 server_base.cc:1034] running on GCE node
I20260928 15:02:10.544957 21929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:10.546420 21929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:10.547768 21929 hybrid_clock.cc:648] HybridClock initialized: now 1790607730547606 us; error 66 us; skew 500 ppm
I20260928 15:02:10.548193 21929 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:02:10 dist-test-slave-j968 krb5kdc[21916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607730, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:10.551530 21929 init.cc:377] Logged in from keytab as kudu/127.31.40.62@KRBTEST.COM (short username kudu)
I20260928 15:02:10.553303 21929 webserver.cc:460] Webserver started at http://127.31.40.62:42833/ using document root <none> and password file <none>
I20260928 15:02:10.554095 21929 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:10.554230 21929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:10.554633 21929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:10.558079 21929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/minicluster-data/master-0/data/instance:
uuid: "1ff57fb559f24d3b883a559b3c20f3d3"
format_stamp: "Formatted at 2026-09-28 15:02:10 on dist-test-slave-j968"
server_key: "98e58cde8adf65ec276d853701b53df4"
server_key_iv: "7b03856772bb10d737a7f4d284ffce0c"
server_key_version: "encryptionkey@0"
I20260928 15:02:10.558903 21929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/minicluster-data/master-0/wal/instance:
uuid: "1ff57fb559f24d3b883a559b3c20f3d3"
format_stamp: "Formatted at 2026-09-28 15:02:10 on dist-test-slave-j968"
server_key: "98e58cde8adf65ec276d853701b53df4"
server_key_iv: "7b03856772bb10d737a7f4d284ffce0c"
server_key_version: "encryptionkey@0"
I20260928 15:02:10.564584 21929 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 15:02:10.568759 21944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:10.570231 21929 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:02:10.570439 21929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/minicluster-data/master-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/minicluster-data/master-0/wal
uuid: "1ff57fb559f24d3b883a559b3c20f3d3"
format_stamp: "Formatted at 2026-09-28 15:02:10 on dist-test-slave-j968"
server_key: "98e58cde8adf65ec276d853701b53df4"
server_key_iv: "7b03856772bb10d737a7f4d284ffce0c"
server_key_version: "encryptionkey@0"
I20260928 15:02:10.570688 21929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:10.589951 21929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:10.594322 21929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:10.594607 21929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:10.614435 21929 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.62:34185
I20260928 15:02:10.614461 21995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.62:34185 every 8 connection(s)
I20260928 15:02:10.615836 21929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
I20260928 15:02:10.618902 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 21929
I20260928 15:02:10.619129 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/minicluster-data/master-0/wal/instance
I20260928 15:02:10.619820 31904 external_mini_cluster.cc:1428] Setting key b2cfa6f4a0f54fc60d47af1d2b9f17de
I20260928 15:02:10.620242 21996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:10.627676 21996 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34185 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.31.40.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.61@KRBTEST.COM" created.
Sep 28 15:02:10 dist-test-slave-j968 krb5kdc[21916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607730, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.62@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 15:02:10.656657 21999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:10.631531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40490 (local address 127.31.40.62:34185)
0928 15:02:10.632562 (+ 1031us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:10.632585 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:10.633066 (+ 481us) server_negotiation.cc:407] Connection header received
0928 15:02:10.635372 (+ 2306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:10.635378 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:10.635638 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:10.635888 (+ 250us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:10.637420 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:10.638817 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:10.640517 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:10.641172 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:10.641426 (+ 254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:10.648014 (+ 6588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:10.648025 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:10.648041 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:10.648082 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:10.651428 (+ 3346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:10.652337 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:10.652345 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:10.652347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:10.652518 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:10.652966 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:10.652976 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:10.652979 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:10.653570 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:10.653816 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:10.654454 (+ 638us) server_negotiation.cc:299] Negotiation successful
0928 15:02:10.654930 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":741,"thread_start_us":492,"threads_started":1}
I20260928 15:02:10.656677 22000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:10.631531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34185 (local address 127.0.0.1:40490)
0928 15:02:10.632881 (+ 1350us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:10.632918 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:10.634941 (+ 2023us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:10.635932 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:10.635940 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:10.636791 (+ 851us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:10.637176 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:10.637197 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:10.638996 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:10.639002 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:10.640352 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:10.640375 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:10.641455 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:10.641463 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:10.641665 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:10.642494 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:10.642520 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:10.647679 (+ 5159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:10.651610 (+ 3931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:10.651624 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:10.651639 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:10.652174 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:10.652638 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:10.652645 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:10.652649 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:10.652842 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:10.653736 (+ 894us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:10.653761 (+ 25us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:10.654018 (+ 257us) client_negotiation.cc:769] Sending connection context
0928 15:02:10.654292 (+ 274us) client_negotiation.cc:240] Negotiation successful
0928 15:02:10.655058 (+ 766us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1109,"mutex_wait_us":221,"spinlock_wait_cycles":17664,"thread_start_us":866,"threads_started":1}
I20260928 15:02:10.660553 21996 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.40.61" port: 36039 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:02:10.661990 22000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:10.661409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.61:36039 (local address 127.0.0.1:40478)
0928 15:02:10.661614 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:10.661846 (+ 232us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.61:36039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260928 15:02:10.662302 21947 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.40.61:36039: connect: Connection refused (error 111)
I20260928 15:02:10.664062 22000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:10.663603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.61:36039 (local address 127.0.0.1:40482)
0928 15:02:10.663798 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:10.663909 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.61:36039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260928 15:02:10.664275 21996 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.40.61:36039: Network error: Client connection negotiation failed: client connection to 127.31.40.61:36039: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.61.keytab.
Entry for principal kudu/127.31.40.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.61.keytab.
I20260928 15:02:10.679783 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.61:36039
--webserver_interface=127.31.40.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.61.keytab
--principal=kudu/127.31.40.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:36313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.40.62:34185,127.31.40.61:36039,127.31.40.60:39205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:02:10.713634 21996 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.40.61" port: 36039 } attempt: 1
I20260928 15:02:10.714775 22000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:10.714328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.61:36039 (local address 127.0.0.1:40486)
0928 15:02:10.714530 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:10.714645 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.61:36039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 15:02:10.715847 22000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:10.715507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.61:36039 (local address 127.0.0.1:40488)
0928 15:02:10.715662 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:10.715724 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.61:36039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 15:02:10.716123 21996 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.40.61:36039: Network error: Client connection negotiation failed: client connection to 127.31.40.61:36039: connect: Connection refused (error 111)
I20260928 15:02:10.784399 21996 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.40.61" port: 36039 } attempt: 2
I20260928 15:02:10.785593 22000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:10.785098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.61:36039 (local address 127.0.0.1:40504)
0928 15:02:10.785364 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:10.785455 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.61:36039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 15:02:10.786633 22000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:10.786271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.61:36039 (local address 127.0.0.1:40514)
0928 15:02:10.786440 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:10.786503 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.61:36039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 15:02:10.786854 21996 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.40.61:36039: Network error: Client connection negotiation failed: client connection to 127.31.40.61:36039: connect: Connection refused (error 111)
I20260928 15:02:10.878119 21996 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.40.61" port: 36039 } attempt: 3
I20260928 15:02:10.879287 22000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:10.878832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.61:36039 (local address 127.0.0.1:40524)
0928 15:02:10.879035 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:10.879135 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.61:36039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 15:02:10.880486 22000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:10.880097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.61:36039 (local address 127.0.0.1:40528)
0928 15:02:10.880276 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:10.880345 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.61:36039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 15:02:10.880785 21996 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.40.61:36039: Network error: Client connection negotiation failed: client connection to 127.31.40.61:36039: connect: Connection refused (error 111)
W20260928 15:02:11.023290 22003 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 15:02:11.023674 22003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:11.023789 22003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:11.023943 22003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 15:02:11.024077 21996 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.40.61" port: 36039 } attempt: 4
I20260928 15:02:11.025435 22000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:11.024757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.61:36039 (local address 127.0.0.1:40542)
0928 15:02:11.025201 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:11.025302 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.61:36039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276}
I20260928 15:02:11.026507 22000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:11.026124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.61:36039 (local address 127.0.0.1:40546)
0928 15:02:11.026286 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:11.026385 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.61:36039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260928 15:02:11.026790 21996 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.40.61:36039: Network error: Client connection negotiation failed: client connection to 127.31.40.61:36039: connect: Connection refused (error 111)
W20260928 15:02:11.031673 22003 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:02:11.031760 22003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:11.031872 22003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:11.031962 22003 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:02:11.032095 22003 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:02:11.042332 22003 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:36313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.40.62:34185,127.31.40.61:36039,127.31.40.60:39205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.61.keytab
--principal=kudu/127.31.40.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.61:36039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.40.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:02:11.043735 22003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:11.045159 22003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:11.057615 22008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:11.057515 22009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:11.057644 22011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:11.058694 22003 server_base.cc:1034] running on GCE node
I20260928 15:02:11.059553 22003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:11.061133 22003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:11.062374 22003 hybrid_clock.cc:648] HybridClock initialized: now 1790607731062303 us; error 36 us; skew 500 ppm
I20260928 15:02:11.062736 22003 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:02:11 dist-test-slave-j968 krb5kdc[21916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607731, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:11.066282 22003 init.cc:377] Logged in from keytab as kudu/127.31.40.61@KRBTEST.COM (short username kudu)
I20260928 15:02:11.068265 22003 webserver.cc:460] Webserver started at http://127.31.40.61:33909/ using document root <none> and password file <none>
I20260928 15:02:11.069046 22003 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:11.069166 22003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:11.069522 22003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:11.072827 22003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/minicluster-data/master-1/data/instance:
uuid: "07b84fc86bd94387aa14b1408a24d257"
format_stamp: "Formatted at 2026-09-28 15:02:11 on dist-test-slave-j968"
server_key: "63a7ea3949601b235d16d74c166ce283"
server_key_iv: "8a699e30dbd829d2a8bc981d7110cb9d"
server_key_version: "encryptionkey@0"
I20260928 15:02:11.073725 22003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/minicluster-data/master-1/wal/instance:
uuid: "07b84fc86bd94387aa14b1408a24d257"
format_stamp: "Formatted at 2026-09-28 15:02:11 on dist-test-slave-j968"
server_key: "63a7ea3949601b235d16d74c166ce283"
server_key_iv: "8a699e30dbd829d2a8bc981d7110cb9d"
server_key_version: "encryptionkey@0"
I20260928 15:02:11.079658 22003 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260928 15:02:11.083454 22018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:11.084998 22003 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:02:11.085218 22003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/minicluster-data/master-1/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/minicluster-data/master-1/wal
uuid: "07b84fc86bd94387aa14b1408a24d257"
format_stamp: "Formatted at 2026-09-28 15:02:11 on dist-test-slave-j968"
server_key: "63a7ea3949601b235d16d74c166ce283"
server_key_iv: "8a699e30dbd829d2a8bc981d7110cb9d"
server_key_version: "encryptionkey@0"
I20260928 15:02:11.085480 22003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:11.104108 22003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:11.108392 22003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:11.108672 22003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:11.127574 22003 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.61:36039
I20260928 15:02:11.127588 22069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.61:36039 every 8 connection(s)
I20260928 15:02:11.129068 22003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/minicluster-data/master-1/data/info.pb
I20260928 15:02:11.133631 22070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:11.137207 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 22003
I20260928 15:02:11.137383 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/minicluster-data/master-1/wal/instance
I20260928 15:02:11.138120 31904 external_mini_cluster.cc:1428] Setting key 498dc013634a3109773cfd663c46c8a9
I20260928 15:02:11.145321 22070 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34185 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:02:11 dist-test-slave-j968 krb5kdc[21916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607731, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.61@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.40.60@KRBTEST.COM; defaulting to no policy
I20260928 15:02:11.172205 21999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:11.149443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40494 (local address 127.31.40.62:34185)
0928 15:02:11.149660 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:11.149666 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:11.151336 (+ 1670us) server_negotiation.cc:407] Connection header received
0928 15:02:11.153507 (+ 2171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:11.153516 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:11.153618 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:11.153727 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:11.156162 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:11.157280 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:11.159900 (+ 2620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:11.160651 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:11.160830 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:11.164750 (+ 3920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:11.164759 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:11.164767 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:11.164818 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:11.168841 (+ 4023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:11.169649 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:11.169656 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:11.169661 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:11.169825 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:11.170276 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:11.170281 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:11.170284 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:11.170696 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:11.170866 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:11.171668 (+ 802us) server_negotiation.cc:299] Negotiation successful
0928 15:02:11.171955 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260928 15:02:11.173345 22072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:11.150133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34185 (local address 127.0.0.1:40494)
0928 15:02:11.151134 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:11.151188 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:11.153183 (+ 1995us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:11.154048 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:11.154063 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:11.154923 (+ 860us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:11.155947 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:11.155981 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:11.157483 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:11.157489 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:11.159684 (+ 2195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:11.159697 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:11.160917 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:11.160925 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:11.161186 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:11.161936 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:11.161962 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:11.164472 (+ 2510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:11.169007 (+ 4535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:11.169015 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:11.169030 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:11.169494 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:11.169956 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:11.169959 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:11.169961 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:11.170143 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:11.170863 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:11.170869 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:11.171220 (+ 351us) client_negotiation.cc:769] Sending connection context
0928 15:02:11.171458 (+ 238us) client_negotiation.cc:240] Negotiation successful
0928 15:02:11.172000 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":394,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.60@KRBTEST.COM" created.
I20260928 15:02:11.175302 22070 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.40.61" port: 36039 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:02:11 dist-test-slave-j968 krb5kdc[21916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607731, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.61@KRBTEST.COM for kudu/127.31.40.61@KRBTEST.COM
I20260928 15:02:11.192636 22072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:11.176211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.61:36039 (local address 127.0.0.1:40562)
0928 15:02:11.176443 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:11.176462 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:11.176589 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:11.177726 (+ 1137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:11.177797 (+ 71us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:11.178160 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:11.178301 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:11.178309 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:11.180332 (+ 2023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:11.180337 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:11.181248 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:11.181262 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:11.182542 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:11.182547 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:11.182671 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:11.183396 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:11.183430 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:11.186309 (+ 2879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:11.190091 (+ 3782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:11.190098 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:11.190102 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:11.190466 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:11.190854 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:11.190860 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:11.190864 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:11.190996 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:11.191613 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:11.191617 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:11.191754 (+ 137us) client_negotiation.cc:769] Sending connection context
0928 15:02:11.191922 (+ 168us) client_negotiation.cc:240] Negotiation successful
0928 15:02:11.192211 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260928 15:02:11.192698 22074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:11.176438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40562 (local address 127.31.40.61:36039)
0928 15:02:11.177181 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:11.177192 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:11.177224 (+ 32us) server_negotiation.cc:407] Connection header received
0928 15:02:11.177293 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:11.177304 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:11.177556 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:11.177705 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:11.178480 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:11.180185 (+ 1705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:11.181397 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:11.182379 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:11.182523 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:11.186520 (+ 3997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:11.186532 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:11.186546 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:11.186580 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:11.189938 (+ 3358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:11.190606 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:11.190618 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:11.190625 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:11.190738 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:11.191100 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:11.191107 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:11.191110 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:11.191469 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:11.191696 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:11.192089 (+ 393us) server_negotiation.cc:299] Negotiation successful
0928 15:02:11.192307 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":549,"thread_start_us":372,"threads_started":1}
I20260928 15:02:11.195533 22070 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.40.60" port: 39205 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:02:11.196965 22072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:11.196481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:39205 (local address 127.0.0.1:53132)
0928 15:02:11.196685 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:11.196813 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.60:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260928 15:02:11.197269 22019 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.40.60:39205: connect: Connection refused (error 111)
I20260928 15:02:11.198966 22072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:11.198620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:39205 (local address 127.0.0.1:53134)
0928 15:02:11.198782 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:11.198858 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.60:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 15:02:11.199209 22070 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.40.60:39205: Network error: Client connection negotiation failed: client connection to 127.31.40.60:39205: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.60.keytab.
Entry for principal kudu/127.31.40.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.60.keytab.
I20260928 15:02:11.211490 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.60:39205
--webserver_interface=127.31.40.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.60.keytab
--principal=kudu/127.31.40.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:36313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.40.62:34185,127.31.40.61:36039,127.31.40.60:39205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:02:11.248518 22070 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.40.60" port: 39205 } attempt: 1
I20260928 15:02:11.249956 22072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:11.249393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:39205 (local address 127.0.0.1:53144)
0928 15:02:11.249618 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:11.249762 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.60:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260928 15:02:11.251104 22072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:11.250647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:39205 (local address 127.0.0.1:53156)
0928 15:02:11.250818 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:11.250943 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.60:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 15:02:11.251361 22070 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.40.60:39205: Network error: Client connection negotiation failed: client connection to 127.31.40.60:39205: connect: Connection refused (error 111)
I20260928 15:02:11.319628 22070 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.40.60" port: 39205 } attempt: 2
I20260928 15:02:11.320919 22072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:11.320439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:39205 (local address 127.0.0.1:53170)
0928 15:02:11.320643 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:11.320739 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.60:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260928 15:02:11.321954 22072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:11.321578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:39205 (local address 127.0.0.1:53180)
0928 15:02:11.321744 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:11.321831 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.60:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 15:02:11.322173 22070 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.40.60:39205: Network error: Client connection negotiation failed: client connection to 127.31.40.60:39205: connect: Connection refused (error 111)
I20260928 15:02:11.326102 21996 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.40.61" port: 36039 } attempt: 5
Sep 28 15:02:11 dist-test-slave-j968 krb5kdc[21916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607730, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.62@KRBTEST.COM for kudu/127.31.40.61@KRBTEST.COM
I20260928 15:02:11.342376 22074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:11.326926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40566 (local address 127.31.40.61:36039)
0928 15:02:11.327090 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:11.327096 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:11.327151 (+ 55us) server_negotiation.cc:407] Connection header received
0928 15:02:11.327341 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:11.327346 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:11.327442 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:11.327558 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:11.328603 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:11.329683 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:11.331148 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:11.331844 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:11.332007 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:11.335929 (+ 3922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:11.335936 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:11.335940 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:11.335982 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:11.339270 (+ 3288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:11.340075 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:11.340081 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:11.340085 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:11.340241 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:11.340783 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:11.340791 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:11.340798 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:11.341109 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:11.341293 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:11.341747 (+ 454us) server_negotiation.cc:299] Negotiation successful
0928 15:02:11.341996 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 15:02:11.342379 22000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:11.326858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.61:36039 (local address 127.0.0.1:40566)
0928 15:02:11.327065 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:11.327092 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:11.327214 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:11.327597 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:11.327601 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:11.328210 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:11.328429 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:11.328440 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:11.329906 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:11.329911 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:11.330972 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:11.330985 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:11.332051 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:11.332056 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:11.332239 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:11.333073 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:11.333093 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:11.335636 (+ 2543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:11.339439 (+ 3803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:11.339443 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:11.339447 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:11.339951 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:11.340424 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:11.340428 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:11.340430 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:11.340613 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:11.341246 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:11.341251 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:11.341430 (+ 179us) client_negotiation.cc:769] Sending connection context
0928 15:02:11.341617 (+ 187us) client_negotiation.cc:240] Negotiation successful
0928 15:02:11.342033 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260928 15:02:11.343560 21996 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.40.60" port: 39205 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:02:11.344906 22000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:11.344371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:39205 (local address 127.0.0.1:53182)
0928 15:02:11.344570 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:11.344675 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.60:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 15:02:11.345999 22000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:11.345613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:39205 (local address 127.0.0.1:53196)
0928 15:02:11.345772 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:11.345870 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.60:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 15:02:11.346274 21996 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.40.60:39205: Network error: Client connection negotiation failed: client connection to 127.31.40.60:39205: connect: Connection refused (error 111)
I20260928 15:02:11.397552 21996 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.40.60" port: 39205 } attempt: 1
I20260928 15:02:11.398891 22000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:11.398386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:39205 (local address 127.0.0.1:53208)
0928 15:02:11.398619 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:11.398713 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.60:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260928 15:02:11.400135 22000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:11.399707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:39205 (local address 127.0.0.1:53210)
0928 15:02:11.399938 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:11.400020 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.60:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260928 15:02:11.400369 21996 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.40.60:39205: Network error: Client connection negotiation failed: client connection to 127.31.40.60:39205: connect: Connection refused (error 111)
I20260928 15:02:11.413416 22070 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.40.60" port: 39205 } attempt: 3
I20260928 15:02:11.414800 22072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:11.414284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:39205 (local address 127.0.0.1:53220)
0928 15:02:11.414539 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:11.414640 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.60:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260928 15:02:11.416102 22072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:11.415566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:39205 (local address 127.0.0.1:53228)
0928 15:02:11.415804 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:11.415984 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.60:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
W20260928 15:02:11.416304 22070 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.40.60:39205: Network error: Client connection negotiation failed: client connection to 127.31.40.60:39205: connect: Connection refused (error 111)
I20260928 15:02:11.468617 21996 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.40.60" port: 39205 } attempt: 2
I20260928 15:02:11.470013 22000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:11.469382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:39205 (local address 127.0.0.1:53230)
0928 15:02:11.469664 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:11.469793 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.60:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260928 15:02:11.471290 22000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:11.470832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:39205 (local address 127.0.0.1:53242)
0928 15:02:11.471015 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:11.471115 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.60:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 15:02:11.471657 21996 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.40.60:39205: Network error: Client connection negotiation failed: client connection to 127.31.40.60:39205: connect: Connection refused (error 111)
I20260928 15:02:11.559544 22070 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.40.60" port: 39205 } attempt: 4
W20260928 15:02:11.560499 22077 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 15:02:11.560863 22077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
I20260928 15:02:11.560932 22072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:11.560312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:39205 (local address 127.0.0.1:53258)
0928 15:02:11.560566 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:11.560654 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.60:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260928 15:02:11.560961 22077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:11.561101 22077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 15:02:11.562137 22072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:11.561783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:39205 (local address 127.0.0.1:53260)
0928 15:02:11.561940 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:11.562023 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.60:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260928 15:02:11.562369 22070 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.40.60:39205: Network error: Client connection negotiation failed: client connection to 127.31.40.60:39205: connect: Connection refused (error 111)
W20260928 15:02:11.569070 22077 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:02:11.569185 22077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:11.569314 22077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:11.569373 22077 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:02:11.569496 22077 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:02:11.577983 21996 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.40.60" port: 39205 } attempt: 3
I20260928 15:02:11.579227 22000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:11.578726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:39205 (local address 127.0.0.1:53268)
0928 15:02:11.578963 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:11.579055 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.60:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260928 15:02:11.580286 22000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:11.579921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:39205 (local address 127.0.0.1:53278)
0928 15:02:11.580106 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:11.580178 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.60:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260928 15:02:11.580644 21996 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.40.60:39205: Network error: Client connection negotiation failed: client connection to 127.31.40.60:39205: connect: Connection refused (error 111)
I20260928 15:02:11.580968 22077 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:36313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.40.62:34185,127.31.40.61:36039,127.31.40.60:39205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.60.keytab
--principal=kudu/127.31.40.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.60:39205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.40.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:02:11.582366 22077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:11.583914 22077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:11.596709 22083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:11.596887 22082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:11.596742 22085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:11.597765 22077 server_base.cc:1034] running on GCE node
I20260928 15:02:11.598594 22077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:11.600092 22077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:11.601325 22077 hybrid_clock.cc:648] HybridClock initialized: now 1790607731601275 us; error 37 us; skew 500 ppm
I20260928 15:02:11.601667 22077 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:02:11 dist-test-slave-j968 krb5kdc[21916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607731, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:11.604978 22077 init.cc:377] Logged in from keytab as kudu/127.31.40.60@KRBTEST.COM (short username kudu)
I20260928 15:02:11.606480 22077 webserver.cc:460] Webserver started at http://127.31.40.60:45497/ using document root <none> and password file <none>
I20260928 15:02:11.607121 22077 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:11.607205 22077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:11.607489 22077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:11.610437 22077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/minicluster-data/master-2/data/instance:
uuid: "b9d7cd96de77439f9dd663b4cfe65c85"
format_stamp: "Formatted at 2026-09-28 15:02:11 on dist-test-slave-j968"
server_key: "c84409b05a6a308fd6e5434db4b98608"
server_key_iv: "553f9b4f17b21dc83d1be43425979cff"
server_key_version: "encryptionkey@0"
I20260928 15:02:11.611152 22077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/minicluster-data/master-2/wal/instance:
uuid: "b9d7cd96de77439f9dd663b4cfe65c85"
format_stamp: "Formatted at 2026-09-28 15:02:11 on dist-test-slave-j968"
server_key: "c84409b05a6a308fd6e5434db4b98608"
server_key_iv: "553f9b4f17b21dc83d1be43425979cff"
server_key_version: "encryptionkey@0"
I20260928 15:02:11.616401 22077 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260928 15:02:11.619956 22092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:11.621302 22077 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:02:11.621476 22077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/minicluster-data/master-2/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/minicluster-data/master-2/wal
uuid: "b9d7cd96de77439f9dd663b4cfe65c85"
format_stamp: "Formatted at 2026-09-28 15:02:11 on dist-test-slave-j968"
server_key: "c84409b05a6a308fd6e5434db4b98608"
server_key_iv: "553f9b4f17b21dc83d1be43425979cff"
server_key_version: "encryptionkey@0"
I20260928 15:02:11.621686 22077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:11.639851 22077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:11.644157 22077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:11.644457 22077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:11.663918 22077 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.60:39205
I20260928 15:02:11.663980 22143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.60:39205 every 8 connection(s)
I20260928 15:02:11.665349 22077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/minicluster-data/master-2/data/info.pb
I20260928 15:02:11.668776 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 22077
I20260928 15:02:11.668972 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607365362981-31904-0/minicluster-data/master-2/wal/instance
I20260928 15:02:11.669437 31904 external_mini_cluster.cc:1428] Setting key e26e239a70401aa5fccf69679e93ac22
I20260928 15:02:11.670176 22144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:11.681569 22144 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34185 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:02:11 dist-test-slave-j968 krb5kdc[21916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.60@KRBTEST.COM
Sep 28 15:02:11 dist-test-slave-j968 krb5kdc[21916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607731, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.60@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 15:02:11.695607 22147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:11.671513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53284 (local address 127.31.40.60:39205)
0928 15:02:11.672429 (+ 916us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:11.672472 (+ 43us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:11.672533 (+ 61us) server_negotiation.cc:407] Connection header received
0928 15:02:11.674396 (+ 1863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:11.674429 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:11.675021 (+ 592us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:11.675531 (+ 510us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:11.677152 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:11.679443 (+ 2291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:11.680833 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:11.682030 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:11.682266 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:11.687036 (+ 4770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:11.687046 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:11.687059 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:11.687106 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:11.690818 (+ 3712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:11.691541 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:11.691547 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:11.691554 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:11.691721 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:11.692232 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:11.692236 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:11.692239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:11.693152 (+ 913us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:11.693471 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:11.693865 (+ 394us) server_negotiation.cc:299] Negotiation successful
0928 15:02:11.694300 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":401,"threads_started":1}
I20260928 15:02:11.701921 22149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:11.684893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:34185 (local address 127.0.0.1:40510)
0928 15:02:11.685814 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:11.685856 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:11.686057 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:11.686489 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:11.686497 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:11.687403 (+ 906us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:11.687840 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:11.687851 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:11.689426 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:11.689438 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:11.690763 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:11.690775 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:11.691547 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:11.691554 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:11.691729 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:11.692405 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:11.692426 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:11.694982 (+ 2556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:11.698646 (+ 3664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:11.698657 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:11.698663 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:11.699269 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:11.699712 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:11.699717 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:11.699720 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:11.700017 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:11.700522 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:11.700530 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:11.700757 (+ 227us) client_negotiation.cc:769] Sending connection context
0928 15:02:11.700961 (+ 204us) client_negotiation.cc:240] Negotiation successful
0928 15:02:11.701344 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"thread_start_us":530,"threads_started":1}
I20260928 15:02:11.702057 22148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:11.684910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40510 (local address 127.31.40.62:34185)
0928 15:02:11.685515 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:11.685522 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:11.686054 (+ 532us) server_negotiation.cc:407] Connection header received
0928 15:02:11.686221 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:11.686225 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:11.686299 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:11.686419 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:11.688039 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:11.689263 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:11.690903 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:11.691352 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:11.691499 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:11.695194 (+ 3695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:11.695201 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:11.695205 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:11.695260 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:11.698493 (+ 3233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:11.699384 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:11.699390 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:11.699397 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:11.699573 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:11.700119 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:11.700122 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:11.700124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:11.700384 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:11.700500 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:11.701116 (+ 616us) server_negotiation.cc:299] Negotiation successful
0928 15:02:11.701395 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":288,"threads_started":1}
I20260928 15:02:11.703667 22144 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.40.61" port: 36039 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:02:11 dist-test-slave-j968 krb5kdc[21916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
Sep 28 15:02:11 dist-test-slave-j968 krb5kdc[21916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607731, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.60@KRBTEST.COM for kudu/127.31.40.61@KRBTEST.COM
I20260928 15:02:11.716107 22150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:11.698772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40524 (local address 127.31.40.62:34185)
0928 15:02:11.699445 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:11.699456 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:11.699479 (+ 23us) server_negotiation.cc:407] Connection header received
0928 15:02:11.699527 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:11.699531 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:11.699595 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:11.699695 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:11.701208 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:11.702548 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:11.704110 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:11.704804 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:11.704929 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:11.709959 (+ 5030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:11.709970 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:11.709975 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:11.710020 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:11.713422 (+ 3402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:11.714098 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:11.714106 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:11.714114 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:11.714236 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:11.714624 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:11.714629 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:11.714631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:11.714896 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:11.715171 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:11.715472 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 15:02:11.715698 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":302,"threads_started":1}
I20260928 15:02:11.719694 22074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:11.704509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40574 (local address 127.31.40.61:36039)
0928 15:02:11.704697 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:11.704703 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:11.704764 (+ 61us) server_negotiation.cc:407] Connection header received
0928 15:02:11.704930 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:11.704937 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:11.705020 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:11.705123 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:11.706020 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:11.707141 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:11.708804 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:11.709476 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:11.709616 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:11.713395 (+ 3779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:11.713402 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:11.713407 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:11.713452 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:11.716776 (+ 3324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:11.717543 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:11.717549 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:11.717553 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:11.717733 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:11.718207 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:11.718213 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:11.718217 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:11.718657 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:11.718805 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:11.719247 (+ 442us) server_negotiation.cc:299] Negotiation successful
0928 15:02:11.719423 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 15:02:11.719944 22149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:11.704486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.61:36039 (local address 127.0.0.1:40574)
0928 15:02:11.704672 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:11.704693 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:11.704807 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:11.705163 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:11.705167 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:11.705711 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:11.705872 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:11.705880 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:11.707308 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:11.707312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:11.708587 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:11.708602 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:11.709671 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:11.709677 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:11.710012 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:11.710640 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:11.710671 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:11.713184 (+ 2513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:11.716908 (+ 3724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:11.716912 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:11.716916 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:11.717426 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:11.717860 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:11.717868 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:11.717874 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:11.718058 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:11.718815 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:11.718818 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:11.718947 (+ 129us) client_negotiation.cc:769] Sending connection context
0928 15:02:11.719130 (+ 183us) client_negotiation.cc:240] Negotiation successful
0928 15:02:11.719440 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 15:02:11.721053 22144 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.40.60" port: 39205 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:02:11 dist-test-slave-j968 krb5kdc[21916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.61@KRBTEST.COM
Sep 28 15:02:11 dist-test-slave-j968 krb5kdc[21916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607731, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.60@KRBTEST.COM for kudu/127.31.40.60@KRBTEST.COM
I20260928 15:02:11.735697 22151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:11.718120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40590 (local address 127.31.40.61:36039)
0928 15:02:11.718799 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:11.718805 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:11.718827 (+ 22us) server_negotiation.cc:407] Connection header received
0928 15:02:11.718874 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:11.718877 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:11.718926 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:11.719001 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:11.720604 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:11.721583 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:11.723365 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:11.724121 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:11.724263 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:11.729037 (+ 4774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:11.729045 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:11.729051 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:11.729100 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:11.732612 (+ 3512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:11.733527 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:11.733532 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:11.733534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:11.733671 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:11.734162 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:11.734168 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:11.734171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:11.734484 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:11.734769 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:11.735073 (+ 304us) server_negotiation.cc:299] Negotiation successful
0928 15:02:11.735310 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":305,"threads_started":1}
I20260928 15:02:11.736264 22147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:11.721860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53286 (local address 127.31.40.60:39205)
0928 15:02:11.722031 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:11.722039 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:11.722239 (+ 200us) server_negotiation.cc:407] Connection header received
0928 15:02:11.722364 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:11.722367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:11.722419 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:11.722488 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:11.723353 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:11.724391 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:11.726233 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:11.726657 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:11.726753 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:11.730627 (+ 3874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:11.730632 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:11.730635 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:11.730664 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:11.733649 (+ 2985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:11.734307 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:11.734314 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:11.734318 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:11.734433 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:11.734779 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:11.734785 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:11.734791 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:11.735048 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:11.735196 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:11.735524 (+ 328us) server_negotiation.cc:299] Negotiation successful
0928 15:02:11.735914 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":97,"server-negotiator.queue_time_us":60}
I20260928 15:02:11.736728 22149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:11.721979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:39205 (local address 127.0.0.1:53286)
0928 15:02:11.722150 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:11.722167 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:11.722268 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:11.722543 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:11.722548 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:11.723025 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:11.723196 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:11.723206 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:11.724517 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:11.724522 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:11.726057 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:11.726075 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:11.726831 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:11.726839 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:11.727047 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:11.728160 (+ 1113us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:11.728183 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:11.730435 (+ 2252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:11.733852 (+ 3417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:11.733856 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:11.733859 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:11.734188 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:11.734542 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:11.734549 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:11.734553 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:11.734679 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:11.735159 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:11.735163 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:11.735306 (+ 143us) client_negotiation.cc:769] Sending connection context
0928 15:02:11.735433 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 15:02:11.736463 (+ 1030us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":497}
I20260928 15:02:11.741056 31904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:02:11.742370 22144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9d7cd96de77439f9dd663b4cfe65c85: Bootstrap starting.
I20260928 15:02:11.745939 22144 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9d7cd96de77439f9dd663b4cfe65c85: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:11.743732 21971 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1ff57fb559f24d3b883a559b3c20f3d3"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40524
I20260928 15:02:11.747079 22144 log.cc:824] T 00000000000000000000000000000000 P b9d7cd96de77439f9dd663b4cfe65c85: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:11.750242 22144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9d7cd96de77439f9dd663b4cfe65c85: No bootstrap required, opened a new log
I20260928 15:02:11.757902 21996 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.40.60" port: 39205 } attempt: 4
I20260928 15:02:11.757715 22144 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9d7cd96de77439f9dd663b4cfe65c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff57fb559f24d3b883a559b3c20f3d3" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34185 } } peers { permanent_uuid: "07b84fc86bd94387aa14b1408a24d257" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 36039 } } peers { permanent_uuid: "b9d7cd96de77439f9dd663b4cfe65c85" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 39205 } }
I20260928 15:02:11.758237 22144 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9d7cd96de77439f9dd663b4cfe65c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d7cd96de77439f9dd663b4cfe65c85, State: Initialized, Role: FOLLOWER
I20260928 15:02:11.758880 22144 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9d7cd96de77439f9dd663b4cfe65c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff57fb559f24d3b883a559b3c20f3d3" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34185 } } peers { permanent_uuid: "07b84fc86bd94387aa14b1408a24d257" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 36039 } } peers { permanent_uuid: "b9d7cd96de77439f9dd663b4cfe65c85" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 39205 } }
I20260928 15:02:11.760020 22153 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9d7cd96de77439f9dd663b4cfe65c85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff57fb559f24d3b883a559b3c20f3d3" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34185 } } peers { permanent_uuid: "07b84fc86bd94387aa14b1408a24d257" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 36039 } } peers { permanent_uuid: "b9d7cd96de77439f9dd663b4cfe65c85" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 39205 } } }
I20260928 15:02:11.760442 22153 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9d7cd96de77439f9dd663b4cfe65c85 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:02:11.761276 22144 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9d7cd96de77439f9dd663b4cfe65c85 [sys.catalog]: configured and running, proceeding with master startup.
Sep 28 15:02:11 dist-test-slave-j968 krb5kdc[21916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607730, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.62@KRBTEST.COM for kudu/127.31.40.60@KRBTEST.COM
W20260928 15:02:11.772379 22164 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b9d7cd96de77439f9dd663b4cfe65c85: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 15:02:11.772536 22164 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 15:02:11.777034 22000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:11.758669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:39205 (local address 127.0.0.1:53298)
0928 15:02:11.758871 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:11.758892 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:11.759006 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:11.759379 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:11.759383 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:11.759799 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:11.760099 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:11.760111 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:11.761607 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:11.761613 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:11.763301 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:11.763333 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:11.764449 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:11.764456 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:11.764670 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:11.765565 (+ 895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:11.765602 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:11.768876 (+ 3274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:11.773707 (+ 4831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:11.773714 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:11.773719 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:11.774290 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:11.774793 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:11.774798 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:11.774804 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:11.774995 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:11.775905 (+ 910us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:11.775911 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:11.776100 (+ 189us) client_negotiation.cc:769] Sending connection context
0928 15:02:11.776254 (+ 154us) client_negotiation.cc:240] Negotiation successful
0928 15:02:11.776608 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 15:02:11.778190 22147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:11.758743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53298 (local address 127.31.40.60:39205)
0928 15:02:11.758960 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:11.758966 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:11.758992 (+ 26us) server_negotiation.cc:407] Connection header received
0928 15:02:11.759143 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:11.759149 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:11.759232 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:11.759329 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:11.760294 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:11.761440 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:11.763500 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:11.764274 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:11.764407 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:11.769149 (+ 4742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:11.769157 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:11.769161 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:11.769205 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:11.773542 (+ 4337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:11.774485 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:11.774492 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:11.774496 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:11.774669 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:11.775127 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:11.775132 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:11.775136 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:11.775630 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:11.775809 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:11.777536 (+ 1727us) server_negotiation.cc:299] Negotiation successful
0928 15:02:11.777786 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 15:02:11.769944 22077 master_runner.cc:186] Error getting master registration for 127.31.40.62:34185: OK, instance_id { permanent_uuid: "1ff57fb559f24d3b883a559b3c20f3d3" instance_seqno: 1790607730608968 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 15:02:11.781944 22077 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 15:02:11.784045 21996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ff57fb559f24d3b883a559b3c20f3d3: Bootstrap starting.
I20260928 15:02:11.787196 21996 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ff57fb559f24d3b883a559b3c20f3d3: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:11.788288 21996 log.cc:824] T 00000000000000000000000000000000 P 1ff57fb559f24d3b883a559b3c20f3d3: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:11.791558 21996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ff57fb559f24d3b883a559b3c20f3d3: No bootstrap required, opened a new log
I20260928 15:02:11.796733 21996 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ff57fb559f24d3b883a559b3c20f3d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff57fb559f24d3b883a559b3c20f3d3" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34185 } } peers { permanent_uuid: "07b84fc86bd94387aa14b1408a24d257" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 36039 } } peers { permanent_uuid: "b9d7cd96de77439f9dd663b4cfe65c85" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 39205 } }
I20260928 15:02:11.797046 21996 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ff57fb559f24d3b883a559b3c20f3d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ff57fb559f24d3b883a559b3c20f3d3, State: Initialized, Role: FOLLOWER
I20260928 15:02:11.797437 21996 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ff57fb559f24d3b883a559b3c20f3d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff57fb559f24d3b883a559b3c20f3d3" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34185 } } peers { permanent_uuid: "07b84fc86bd94387aa14b1408a24d257" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 36039 } } peers { permanent_uuid: "b9d7cd96de77439f9dd663b4cfe65c85" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 39205 } }
I20260928 15:02:11.798529 22167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ff57fb559f24d3b883a559b3c20f3d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff57fb559f24d3b883a559b3c20f3d3" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34185 } } peers { permanent_uuid: "07b84fc86bd94387aa14b1408a24d257" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 36039 } } peers { permanent_uuid: "b9d7cd96de77439f9dd663b4cfe65c85" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 39205 } } }
I20260928 15:02:11.799225 21996 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ff57fb559f24d3b883a559b3c20f3d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:02:11.799364 22167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ff57fb559f24d3b883a559b3c20f3d3 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:02:11.799427 21971 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ff57fb559f24d3b883a559b3c20f3d3 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 15:02:11.799633 21971 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ff57fb559f24d3b883a559b3c20f3d3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:11.803936 21971 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ff57fb559f24d3b883a559b3c20f3d3 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff57fb559f24d3b883a559b3c20f3d3" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34185 } } peers { permanent_uuid: "07b84fc86bd94387aa14b1408a24d257" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 36039 } } peers { permanent_uuid: "b9d7cd96de77439f9dd663b4cfe65c85" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 39205 } }
I20260928 15:02:11.807075 22119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ff57fb559f24d3b883a559b3c20f3d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b9d7cd96de77439f9dd663b4cfe65c85"
I20260928 15:02:11.807680 22119 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9d7cd96de77439f9dd663b4cfe65c85 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:11.810357 21971 leader_election.cc:288] T 00000000000000000000000000000000 P 1ff57fb559f24d3b883a559b3c20f3d3 [CANDIDATE]: Term 1 election: Requested vote from peers 07b84fc86bd94387aa14b1408a24d257 (127.31.40.61:36039), b9d7cd96de77439f9dd663b4cfe65c85 (127.31.40.60:39205)
I20260928 15:02:11.812163 22119 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b9d7cd96de77439f9dd663b4cfe65c85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ff57fb559f24d3b883a559b3c20f3d3 in term 1.
I20260928 15:02:11.812924 21945 leader_election.cc:302] T 00000000000000000000000000000000 P 1ff57fb559f24d3b883a559b3c20f3d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ff57fb559f24d3b883a559b3c20f3d3, b9d7cd96de77439f9dd663b4cfe65c85; no voters:
I20260928 15:02:11.813521 22167 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ff57fb559f24d3b883a559b3c20f3d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:11.823217 22167 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ff57fb559f24d3b883a559b3c20f3d3 [term 1 LEADER]: Becoming Leader. State: Replica: 1ff57fb559f24d3b883a559b3c20f3d3, State: Running, Role: LEADER
I20260928 15:02:11.824048 22167 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ff57fb559f24d3b883a559b3c20f3d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff57fb559f24d3b883a559b3c20f3d3" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34185 } } peers { permanent_uuid: "07b84fc86bd94387aa14b1408a24d257" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 36039 } } peers { permanent_uuid: "b9d7cd96de77439f9dd663b4cfe65c85" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 39205 } }
I20260928 15:02:11.830207 22178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ff57fb559f24d3b883a559b3c20f3d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ff57fb559f24d3b883a559b3c20f3d3. Latest consensus state: current_term: 1 leader_uuid: "1ff57fb559f24d3b883a559b3c20f3d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff57fb559f24d3b883a559b3c20f3d3" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34185 } } peers { permanent_uuid: "07b84fc86bd94387aa14b1408a24d257" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 36039 } } peers { permanent_uuid: "b9d7cd96de77439f9dd663b4cfe65c85" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 39205 } } }
I20260928 15:02:11.830854 22178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ff57fb559f24d3b883a559b3c20f3d3 [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:11.835340 22181 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260928 15:02:11.847561 21950 tablet.cc:2369] T 00000000000000000000000000000000 P 1ff57fb559f24d3b883a559b3c20f3d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:02:11.852229 22181 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:02:11.861600 22070 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.40.60" port: 39205 } attempt: 5
Sep 28 15:02:11 dist-test-slave-j968 krb5kdc[21916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607731, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.61@KRBTEST.COM for kudu/127.31.40.60@KRBTEST.COM
I20260928 15:02:11.887151 22119 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b9d7cd96de77439f9dd663b4cfe65c85 [term 1 FOLLOWER]: Refusing update from remote peer 1ff57fb559f24d3b883a559b3c20f3d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:02:11.888550 22178 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1ff57fb559f24d3b883a559b3c20f3d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d7cd96de77439f9dd663b4cfe65c85" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:02:11.881143 21929 master_runner.cc:186] Error getting master registration for 127.31.40.61:36039: OK, instance_id { permanent_uuid: "07b84fc86bd94387aa14b1408a24d257" instance_seqno: 1790607731122251 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
W20260928 15:02:11.896608 22098 tablet.cc:2369] T 00000000000000000000000000000000 P b9d7cd96de77439f9dd663b4cfe65c85: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:02:11.896768 21929 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 15:02:11.901912 22147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:11.864051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53306 (local address 127.31.40.60:39205)
0928 15:02:11.864274 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:11.864284 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:11.864311 (+ 27us) server_negotiation.cc:407] Connection header received
0928 15:02:11.864606 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:11.864617 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:11.864741 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:11.864856 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:11.866437 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:11.868042 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:11.870599 (+ 2557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:11.871605 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:11.871778 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:11.878032 (+ 6254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:11.878043 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:11.878051 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:11.878140 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:11.883415 (+ 5275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:11.895974 (+ 12559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:11.895980 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:11.895983 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:11.896194 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:11.899238 (+ 3044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:11.899244 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:11.899248 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:11.899785 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:11.899999 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:11.900936 (+ 937us) server_negotiation.cc:299] Negotiation successful
0928 15:02:11.901297 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 15:02:11.903120 22072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:11.863187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:39205 (local address 127.0.0.1:53306)
0928 15:02:11.863706 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:11.863735 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:11.864393 (+ 658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:11.865122 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:11.865129 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:11.865875 (+ 746us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:11.866193 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:11.866211 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:11.868264 (+ 2053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:11.868274 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:11.870359 (+ 2085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:11.870383 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:11.872088 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:11.872096 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:11.872391 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:11.873412 (+ 1021us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:11.873513 (+ 101us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:11.877704 (+ 4191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:11.883689 (+ 5985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:11.883697 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:11.883704 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:11.884471 (+ 767us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:11.897642 (+ 13171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:11.897651 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:11.897658 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:11.897901 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:11.900112 (+ 2211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:11.900120 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:11.900360 (+ 240us) client_negotiation.cc:769] Sending connection context
0928 15:02:11.902116 (+ 1756us) client_negotiation.cc:240] Negotiation successful
0928 15:02:11.902517 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324}
I20260928 15:02:11.904731 22153 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9d7cd96de77439f9dd663b4cfe65c85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ff57fb559f24d3b883a559b3c20f3d3. Latest consensus state: current_term: 1 leader_uuid: "1ff57fb559f24d3b883a559b3c20f3d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff57fb559f24d3b883a559b3c20f3d3" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34185 } } peers { permanent_uuid: "07b84fc86bd94387aa14b1408a24d257" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 36039 } } peers { permanent_uuid: "b9d7cd96de77439f9dd663b4cfe65c85" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 39205 } } }
I20260928 15:02:11.905131 22153 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9d7cd96de77439f9dd663b4cfe65c85 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:02:11.908668 22167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ff57fb559f24d3b883a559b3c20f3d3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1ff57fb559f24d3b883a559b3c20f3d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff57fb559f24d3b883a559b3c20f3d3" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34185 } } peers { permanent_uuid: "07b84fc86bd94387aa14b1408a24d257" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 36039 } } peers { permanent_uuid: "b9d7cd96de77439f9dd663b4cfe65c85" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 39205 } } }
I20260928 15:02:11.909096 22167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ff57fb559f24d3b883a559b3c20f3d3 [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:11.909473 22186 mvcc.cc:205] Tried to move back new op lower bound from 7334329269716738048 to 7334329269560295424. Current Snapshot: MvccSnapshot[applied={T|T < 7334329269716738048}]
I20260928 15:02:11.909603 22188 mvcc.cc:205] Tried to move back new op lower bound from 7334329269716738048 to 7334329269560295424. Current Snapshot: MvccSnapshot[applied={T|T < 7334329269716738048}]
I20260928 15:02:11.911502 22070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07b84fc86bd94387aa14b1408a24d257: Bootstrap starting.
I20260928 15:02:11.913368 22181 catalog_manager.cc:1347] Generated new cluster ID: db1536d6b47849289e06a0c54ab746c4
I20260928 15:02:11.913486 22181 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:02:11.915699 22070 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07b84fc86bd94387aa14b1408a24d257: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:11.926219 22070 log.cc:824] T 00000000000000000000000000000000 P 07b84fc86bd94387aa14b1408a24d257: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:11.927106 22153 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9d7cd96de77439f9dd663b4cfe65c85 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1ff57fb559f24d3b883a559b3c20f3d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff57fb559f24d3b883a559b3c20f3d3" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34185 } } peers { permanent_uuid: "07b84fc86bd94387aa14b1408a24d257" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 36039 } } peers { permanent_uuid: "b9d7cd96de77439f9dd663b4cfe65c85" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 39205 } } }
I20260928 15:02:11.927399 22153 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9d7cd96de77439f9dd663b4cfe65c85 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:02:11.943610 22070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07b84fc86bd94387aa14b1408a24d257: No bootstrap required, opened a new log
I20260928 15:02:11.952649 22181 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:02:11.955302 22070 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07b84fc86bd94387aa14b1408a24d257 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff57fb559f24d3b883a559b3c20f3d3" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34185 } } peers { permanent_uuid: "07b84fc86bd94387aa14b1408a24d257" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 36039 } } peers { permanent_uuid: "b9d7cd96de77439f9dd663b4cfe65c85" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 39205 } }
I20260928 15:02:11.956511 22181 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:02:11.955785 22070 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07b84fc86bd94387aa14b1408a24d257 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07b84fc86bd94387aa14b1408a24d257, State: Initialized, Role: FOLLOWER
I20260928 15:02:11.972539 22070 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07b84fc86bd94387aa14b1408a24d257 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff57fb559f24d3b883a559b3c20f3d3" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34185 } } peers { permanent_uuid: "07b84fc86bd94387aa14b1408a24d257" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 36039 } } peers { permanent_uuid: "b9d7cd96de77439f9dd663b4cfe65c85" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 39205 } }
I20260928 15:02:11.974126 22196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07b84fc86bd94387aa14b1408a24d257 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff57fb559f24d3b883a559b3c20f3d3" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34185 } } peers { permanent_uuid: "07b84fc86bd94387aa14b1408a24d257" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 36039 } } peers { permanent_uuid: "b9d7cd96de77439f9dd663b4cfe65c85" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 39205 } } }
I20260928 15:02:11.975184 22196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07b84fc86bd94387aa14b1408a24d257 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:02:11.975512 22070 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07b84fc86bd94387aa14b1408a24d257 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:02:11.975533 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ff57fb559f24d3b883a559b3c20f3d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "07b84fc86bd94387aa14b1408a24d257"
I20260928 15:02:11.977768 22044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07b84fc86bd94387aa14b1408a24d257 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:11.981405 22044 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 07b84fc86bd94387aa14b1408a24d257 [term 1 FOLLOWER]: Refusing update from remote peer 1ff57fb559f24d3b883a559b3c20f3d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:02:11.982347 22178 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1ff57fb559f24d3b883a559b3c20f3d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07b84fc86bd94387aa14b1408a24d257" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 36039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:02:11.985445 22181 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ff57fb559f24d3b883a559b3c20f3d3: Generated new TSK 0
I20260928 15:02:11.986887 22181 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:02:12.001788 22150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:11.913301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40528 (local address 127.31.40.62:34185)
0928 15:02:11.913498 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:11.913504 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:11.948206 (+ 34702us) server_negotiation.cc:407] Connection header received
0928 15:02:11.949918 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:11.949924 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:11.950015 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:11.950124 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:11.959105 (+ 8981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:11.960300 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:11.965773 (+ 5473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:11.966593 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:11.966702 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:11.981943 (+ 15241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:11.981951 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:11.981957 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:11.982002 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:11.986037 (+ 4035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:11.992300 (+ 6263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:11.992307 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:11.992311 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:11.992475 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:11.993609 (+ 1134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:11.993617 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:11.993622 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:11.994052 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:11.994325 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:12.001020 (+ 6695us) server_negotiation.cc:299] Negotiation successful
0928 15:02:12.001352 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":62}
I20260928 15:02:12.015653 22199 mvcc.cc:205] Tried to move back new op lower bound from 7334329269716738048 to 7334329269560295424. Current Snapshot: MvccSnapshot[applied={T|T < 7334329269716738048}]
I20260928 15:02:12.019589 22151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:11.920186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40604 (local address 127.31.40.61:36039)
0928 15:02:11.920407 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:11.920413 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:11.950705 (+ 30292us) server_negotiation.cc:407] Connection header received
0928 15:02:11.953120 (+ 2415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:11.953127 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:11.953218 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:11.953316 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:11.962349 (+ 9033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:11.963529 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:11.970967 (+ 7438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:11.971705 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:11.971844 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:11.987007 (+ 15163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:11.987015 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:11.987020 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:11.987065 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:11.991351 (+ 4286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:11.992970 (+ 1619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:11.992975 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:11.992979 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:11.993126 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:11.994840 (+ 1714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:11.994846 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:11.994850 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:11.995262 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:11.995494 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:12.018890 (+ 23396us) server_negotiation.cc:299] Negotiation successful
0928 15:02:12.019198 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 15:02:12.025323 22196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07b84fc86bd94387aa14b1408a24d257 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ff57fb559f24d3b883a559b3c20f3d3. Latest consensus state: current_term: 1 leader_uuid: "1ff57fb559f24d3b883a559b3c20f3d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff57fb559f24d3b883a559b3c20f3d3" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34185 } } peers { permanent_uuid: "07b84fc86bd94387aa14b1408a24d257" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 36039 } } peers { permanent_uuid: "b9d7cd96de77439f9dd663b4cfe65c85" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 39205 } } }
I20260928 15:02:12.025736 22196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07b84fc86bd94387aa14b1408a24d257 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:02:12.026192 22198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07b84fc86bd94387aa14b1408a24d257 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1ff57fb559f24d3b883a559b3c20f3d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff57fb559f24d3b883a559b3c20f3d3" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34185 } } peers { permanent_uuid: "07b84fc86bd94387aa14b1408a24d257" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 36039 } } peers { permanent_uuid: "b9d7cd96de77439f9dd663b4cfe65c85" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 39205 } } }
I20260928 15:02:12.026650 22198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07b84fc86bd94387aa14b1408a24d257 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:02:12.027134 22147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:11.930654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53320 (local address 127.31.40.60:39205)
0928 15:02:11.930877 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:11.930884 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:11.956158 (+ 25274us) server_negotiation.cc:407] Connection header received
0928 15:02:11.956782 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:11.956788 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:11.956871 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:11.956947 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:11.966929 (+ 9982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:11.968085 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:11.975672 (+ 7587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:11.976418 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:11.976528 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:11.996057 (+ 19529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:11.996066 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:11.996071 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:11.996117 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:11.999970 (+ 3853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:12.023593 (+ 23623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:12.023600 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:12.023605 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:12.023824 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:12.024656 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:12.024662 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:12.024665 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:12.025109 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:12.025434 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:12.026481 (+ 1047us) server_negotiation.cc:299] Negotiation successful
0928 15:02:12.026761 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 15:02:12.027195 22045 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 07b84fc86bd94387aa14b1408a24d257 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ff57fb559f24d3b883a559b3c20f3d3 for term 1 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260928 15:02:12.111967 22214 catalog_manager.cc:1259] Loaded cluster ID: db1536d6b47849289e06a0c54ab746c4
I20260928 15:02:12.112100 22214 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 07b84fc86bd94387aa14b1408a24d257: loading cluster ID for follower catalog manager: success
I20260928 15:02:12.116056 22214 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 07b84fc86bd94387aa14b1408a24d257: acquiring CA information for follower catalog manager: success
I20260928 15:02:12.118667 22214 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 07b84fc86bd94387aa14b1408a24d257: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 15:02:12.127585 22150 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:02:12.124948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40538 (local address 127.31.40.62:34185)
0928 15:02:12.125311 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:12.125318 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:12.125339 (+ 21us) server_negotiation.cc:407] Connection header received
0928 15:02:12.125414 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:12.125421 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:12.125539 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:12.125648 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:12.127264 (+ 1616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40538: BlockingRecv error: recv got EOF from 127.0.0.1:40538 (error 108)
Metrics: {"server-negotiator.queue_time_us":207}
W20260928 15:02:12.149857 22147 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:02:12.125258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53336 (local address 127.31.40.60:39205)
0928 15:02:12.125542 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:12.125548 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:12.136238 (+ 10690us) server_negotiation.cc:407] Connection header received
0928 15:02:12.148301 (+ 12063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:12.148308 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:12.148415 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:12.148494 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:12.149593 (+ 1099us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53336: BlockingRecv error: recv got EOF from 127.0.0.1:53336 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260928 15:02:12.152365 22151 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:02:12.126757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40610 (local address 127.31.40.61:36039)
0928 15:02:12.126947 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:12.126952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:12.150685 (+ 23733us) server_negotiation.cc:407] Connection header received
0928 15:02:12.150943 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:12.150950 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:12.151036 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:12.151106 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:12.152129 (+ 1023us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40610: BlockingRecv error: recv got EOF from 127.0.0.1:40610 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 15:02:12.228480 22151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:12.204260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40614 (local address 127.31.40.61:36039)
0928 15:02:12.204490 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:12.204498 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:12.220512 (+ 16014us) server_negotiation.cc:407] Connection header received
0928 15:02:12.220840 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:12.220848 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:12.221080 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:12.221239 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:12.222051 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:12.223224 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:12.225329 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:12.225986 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:12.226071 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:12.226875 (+ 804us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:12.227589 (+ 714us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:12.227700 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:12.227991 (+ 291us) server_negotiation.cc:299] Negotiation successful
0928 15:02:12.228086 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260928 15:02:12.276608 22147 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:02:12.263224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53338 (local address 127.31.40.60:39205)
0928 15:02:12.263431 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:12.263437 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:12.274090 (+ 10653us) server_negotiation.cc:407] Connection header received
0928 15:02:12.274777 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:12.274784 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:12.274928 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:12.275013 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:12.276346 (+ 1333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53338: BlockingRecv error: recv got EOF from 127.0.0.1:53338 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 15:02:12.328454 22151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:12.312903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40628 (local address 127.31.40.61:36039)
0928 15:02:12.313112 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:12.313120 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:12.320621 (+ 7501us) server_negotiation.cc:407] Connection header received
0928 15:02:12.321205 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:12.321212 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:12.321311 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:12.321382 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:12.322262 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:12.323437 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:12.325772 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:12.326592 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:12.326709 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:12.327469 (+ 760us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:12.327620 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:12.327684 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:12.328009 (+ 325us) server_negotiation.cc:299] Negotiation successful
0928 15:02:12.328104 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260928 15:02:12.390244 22147 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:02:12.365263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53342 (local address 127.31.40.60:39205)
0928 15:02:12.365451 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:12.365457 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:12.388578 (+ 23121us) server_negotiation.cc:407] Connection header received
0928 15:02:12.388841 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:12.388849 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:12.388975 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:12.389085 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:12.390002 (+ 917us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53342: BlockingRecv error: recv got EOF from 127.0.0.1:53342 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 15:02:12.447359 22151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:12.430511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40642 (local address 127.31.40.61:36039)
0928 15:02:12.430930 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:12.430939 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:12.438969 (+ 8030us) server_negotiation.cc:407] Connection header received
0928 15:02:12.439516 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:12.439524 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:12.439619 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:12.439729 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:12.441420 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:12.442691 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:12.444888 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:12.445599 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:12.445699 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:12.446430 (+ 731us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:12.446609 (+ 179us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:12.446685 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:12.446931 (+ 246us) server_negotiation.cc:299] Negotiation successful
0928 15:02:12.447023 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260928 15:02:12.474188 22178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ff57fb559f24d3b883a559b3c20f3d3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1ff57fb559f24d3b883a559b3c20f3d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff57fb559f24d3b883a559b3c20f3d3" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 34185 } } peers { permanent_uuid: "07b84fc86bd94387aa14b1408a24d257" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 36039 } } peers { permanent_uuid: "b9d7cd96de77439f9dd663b4cfe65c85" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 39205 } } }
I20260928 15:02:12.474658 22178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ff57fb559f24d3b883a559b3c20f3d3 [sys.catalog]: This master's current role is: LEADER
W20260928 15:02:12.494760 22147 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:02:12.484512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53356 (local address 127.31.40.60:39205)
0928 15:02:12.484701 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:12.484708 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:12.493230 (+ 8522us) server_negotiation.cc:407] Connection header received
0928 15:02:12.493394 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:12.493401 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:12.493502 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:12.493619 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:12.494481 (+ 862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53356: BlockingRecv error: recv got EOF from 127.0.0.1:53356 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 15:02:12.509954 22151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:12.502624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40654 (local address 127.31.40.61:36039)
0928 15:02:12.502878 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:12.502886 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:12.503293 (+ 407us) server_negotiation.cc:407] Connection header received
0928 15:02:12.503485 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:12.503491 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:12.503593 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:12.503700 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:12.504207 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:12.505416 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:12.507555 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:12.508260 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:12.508391 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:12.508855 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:12.509026 (+ 171us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:12.509112 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:12.509323 (+ 211us) server_negotiation.cc:299] Negotiation successful
0928 15:02:12.509415 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
W20260928 15:02:12.518790 22147 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:02:12.517165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53364 (local address 127.31.40.60:39205)
0928 15:02:12.517438 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:12.517444 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:12.517464 (+ 20us) server_negotiation.cc:407] Connection header received
0928 15:02:12.517534 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:12.517539 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:12.517625 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:12.517727 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:12.518578 (+ 851us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53364: BlockingRecv error: recv got EOF from 127.0.0.1:53364 (error 108)
Metrics: {"server-negotiator.queue_time_us":124}
I20260928 15:02:12.540091 22151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:12.533329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40658 (local address 127.31.40.61:36039)
0928 15:02:12.533519 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:12.533527 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:12.534051 (+ 524us) server_negotiation.cc:407] Connection header received
0928 15:02:12.534254 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:12.534260 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:12.534350 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:12.534444 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:12.534945 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:12.536156 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:12.538002 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:12.538667 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:12.538787 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:12.539232 (+ 445us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:12.539378 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:12.539468 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:12.539626 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 15:02:12.539719 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260928 15:02:12.548005 22147 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:02:12.545973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53370 (local address 127.31.40.60:39205)
0928 15:02:12.546147 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:12.546152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:12.546637 (+ 485us) server_negotiation.cc:407] Connection header received
0928 15:02:12.546778 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:12.546783 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:12.546882 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:12.546988 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:12.547712 (+ 724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53370: BlockingRecv error: recv got EOF from 127.0.0.1:53370 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 15:02:12.574545 22151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:12.567765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40660 (local address 127.31.40.61:36039)
0928 15:02:12.568043 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:12.568051 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:12.568452 (+ 401us) server_negotiation.cc:407] Connection header received
0928 15:02:12.568635 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:12.568642 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:12.568755 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:12.568894 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:12.569331 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:12.570809 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:12.572327 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:12.573083 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:12.573235 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:12.573676 (+ 441us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:12.573834 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:12.573929 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:12.574115 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 15:02:12.574211 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
W20260928 15:02:12.583058 22147 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:02:12.580928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53380 (local address 127.31.40.60:39205)
0928 15:02:12.581107 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:12.581111 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:12.581636 (+ 525us) server_negotiation.cc:407] Connection header received
0928 15:02:12.581812 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:12.581820 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:12.581938 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:12.582059 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:12.582809 (+ 750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53380: BlockingRecv error: recv got EOF from 127.0.0.1:53380 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 15:02:12.624152 22151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:12.618472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40664 (local address 127.31.40.61:36039)
0928 15:02:12.618730 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:12.618739 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:12.619306 (+ 567us) server_negotiation.cc:407] Connection header received
0928 15:02:12.619555 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:12.619560 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:12.619631 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:12.619735 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:12.620230 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:12.621032 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:12.622357 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:12.622799 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:12.622943 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:12.623355 (+ 412us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:12.623460 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:12.623555 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:12.623704 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 15:02:12.623783 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
W20260928 15:02:12.637965 22147 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:02:12.634605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53396 (local address 127.31.40.60:39205)
0928 15:02:12.634807 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:12.634814 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:12.636460 (+ 1646us) server_negotiation.cc:407] Connection header received
0928 15:02:12.636705 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:12.636715 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:12.636791 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:12.636907 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:12.637666 (+ 759us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53396: BlockingRecv error: recv got EOF from 127.0.0.1:53396 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 15:02:12.712946 22151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:12.706378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40678 (local address 127.31.40.61:36039)
0928 15:02:12.706653 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:12.706660 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:12.707054 (+ 394us) server_negotiation.cc:407] Connection header received
0928 15:02:12.707155 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:12.707161 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:12.707255 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:12.707372 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:12.707828 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:12.709200 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:12.710763 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:12.711444 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:12.711572 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:12.712112 (+ 540us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:12.712270 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:12.712358 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:12.712500 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 15:02:12.712587 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260928 15:02:12.728459 22147 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:02:12.726141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53410 (local address 127.31.40.60:39205)
0928 15:02:12.726347 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:12.726354 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:12.726902 (+ 548us) server_negotiation.cc:407] Connection header received
0928 15:02:12.727077 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:12.727083 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:12.727177 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:12.727291 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:12.728242 (+ 951us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53410: BlockingRecv error: recv got EOF from 127.0.0.1:53410 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 15:02:12.774935 22164 catalog_manager.cc:1259] Loaded cluster ID: db1536d6b47849289e06a0c54ab746c4
I20260928 15:02:12.775038 22164 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b9d7cd96de77439f9dd663b4cfe65c85: loading cluster ID for follower catalog manager: success
I20260928 15:02:12.778084 22164 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b9d7cd96de77439f9dd663b4cfe65c85: acquiring CA information for follower catalog manager: success
I20260928 15:02:12.780097 22164 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b9d7cd96de77439f9dd663b4cfe65c85: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 15:02:12.865480 22151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:12.859957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40682 (local address 127.31.40.61:36039)
0928 15:02:12.860224 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:12.860231 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:12.860555 (+ 324us) server_negotiation.cc:407] Connection header received
0928 15:02:12.860792 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:12.860796 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:12.860908 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:12.861035 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:12.861451 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:12.862299 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:12.863655 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:12.864200 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:12.864358 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:12.864755 (+ 397us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:12.864867 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:12.864968 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:12.865111 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 15:02:12.865191 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 15:02:12.877830 22147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:12.871550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53418 (local address 127.31.40.60:39205)
0928 15:02:12.871784 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:12.871789 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:12.872054 (+ 265us) server_negotiation.cc:407] Connection header received
0928 15:02:12.872192 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:12.872196 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:12.872361 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:12.872461 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:12.873005 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:12.873794 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:12.875342 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:12.875843 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:12.876008 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:12.876649 (+ 641us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:12.877228 (+ 579us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:12.877368 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:12.877479 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 15:02:12.877548 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 15:02:12.903836 22151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:12.885469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40690 (local address 127.31.40.61:36039)
0928 15:02:12.885716 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:12.885725 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:12.897336 (+ 11611us) server_negotiation.cc:407] Connection header received
0928 15:02:12.897502 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:12.897512 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:12.897671 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:12.897811 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:12.898687 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:12.899916 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:12.901713 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:12.902186 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:12.902365 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:12.902956 (+ 591us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:12.903105 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:12.903197 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:12.903395 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 15:02:12.903491 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 15:02:12.905267 22147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:12.885252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53430 (local address 127.31.40.60:39205)
0928 15:02:12.885663 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:12.885672 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:12.898271 (+ 12599us) server_negotiation.cc:407] Connection header received
0928 15:02:12.898550 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:12.898560 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:12.898669 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:12.898770 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:12.899487 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:12.900769 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:12.902974 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:12.903738 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:12.903940 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:12.904428 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:12.904528 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:12.904625 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:12.904894 (+ 269us) server_negotiation.cc:299] Negotiation successful
0928 15:02:12.904974 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204}
I20260928 15:02:12.906992 22317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:12.884629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40544 (local address 127.31.40.62:34185)
0928 15:02:12.898448 (+ 13819us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:12.898456 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:12.898478 (+ 22us) server_negotiation.cc:407] Connection header received
0928 15:02:12.898616 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:12.898622 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:12.898914 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:12.899131 (+ 217us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:12.899796 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:12.901053 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:12.904396 (+ 3343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:12.905083 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:12.905209 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:12.906030 (+ 821us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:12.906381 (+ 351us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:12.906506 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:12.906625 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 15:02:12.906720 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13665,"thread_start_us":13551,"threads_started":1}
I20260928 15:02:12.912914 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 21929
I20260928 15:02:12.914146 21991 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:02:13.028225 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 21929
I20260928 15:02:13.050446 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 22003
I20260928 15:02:13.051631 22065 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:02:13.160892 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 22003
I20260928 15:02:13.182583 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 22077
I20260928 15:02:13.183781 22139 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:02:13.292611 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 22077
2026-09-28T15:02:13Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4568 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:02:13 dist-test-slave-j968 krb5kdc[22326](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:02:13 dist-test-slave-j968 krb5kdc[22326](info): set up 2 sockets
Sep 28 15:02:13 dist-test-slave-j968 krb5kdc[22326](info): commencing operation
krb5kdc: starting...
W20260928 15:02:14.437626 31904 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/krb5kdc/test-user.keytab.
Sep 28 15:02:14 dist-test-slave-j968 krb5kdc[22326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:02:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:02:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.40.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab.
Entry for principal kudu/127.31.40.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab.
I20260928 15:02:14.720669 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:33005
--webserver_interface=127.31.40.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:38035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.40.62:33005,127.31.40.61:45263,127.31.40.60:42911
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:15.074098 22339 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 15:02:15.074517 22339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:15.074651 22339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:15.074797 22339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:15.082430 22339 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:02:15.082567 22339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:15.082674 22339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:15.082770 22339 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:02:15.082875 22339 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:02:15.092959 22339 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:38035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.40.62:33005,127.31.40.61:45263,127.31.40.60:42911
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.62.keytab
--principal=kudu/127.31.40.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.62:33005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.40.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:02:15.094336 22339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:15.095690 22339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:15.107936 22344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:15.107991 22347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:15.108027 22345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:15.109104 22339 server_base.cc:1034] running on GCE node
I20260928 15:02:15.109970 22339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:15.111455 22339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:15.112712 22339 hybrid_clock.cc:648] HybridClock initialized: now 1790607735112636 us; error 50 us; skew 500 ppm
I20260928 15:02:15.113078 22339 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:02:15 dist-test-slave-j968 krb5kdc[22326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607735, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:15.122787 22339 init.cc:377] Logged in from keytab as kudu/127.31.40.62@KRBTEST.COM (short username kudu)
I20260928 15:02:15.124612 22339 webserver.cc:460] Webserver started at http://127.31.40.62:35263/ using document root <none> and password file <none>
I20260928 15:02:15.125440 22339 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:15.125590 22339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:15.125954 22339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:15.129402 22339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/minicluster-data/master-0/data/instance:
uuid: "f8eb0cd40dc74a50bd040a59f1372fbe"
format_stamp: "Formatted at 2026-09-28 15:02:15 on dist-test-slave-j968"
server_key: "6ba2a4db6c02eb935b73bc2f75584c7d"
server_key_iv: "70b812ff1386be67df5d78995579eba9"
server_key_version: "encryptionkey@0"
I20260928 15:02:15.130344 22339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/minicluster-data/master-0/wal/instance:
uuid: "f8eb0cd40dc74a50bd040a59f1372fbe"
format_stamp: "Formatted at 2026-09-28 15:02:15 on dist-test-slave-j968"
server_key: "6ba2a4db6c02eb935b73bc2f75584c7d"
server_key_iv: "70b812ff1386be67df5d78995579eba9"
server_key_version: "encryptionkey@0"
I20260928 15:02:15.136360 22339 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260928 15:02:15.140391 22354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:15.142163 22339 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 15:02:15.142385 22339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/minicluster-data/master-0/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/minicluster-data/master-0/wal
uuid: "f8eb0cd40dc74a50bd040a59f1372fbe"
format_stamp: "Formatted at 2026-09-28 15:02:15 on dist-test-slave-j968"
server_key: "6ba2a4db6c02eb935b73bc2f75584c7d"
server_key_iv: "70b812ff1386be67df5d78995579eba9"
server_key_version: "encryptionkey@0"
I20260928 15:02:15.142647 22339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:15.164201 22339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:15.169134 22339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:15.169453 22339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:15.190435 22339 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.62:33005
I20260928 15:02:15.190455 22405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.62:33005 every 8 connection(s)
I20260928 15:02:15.192039 22339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/minicluster-data/master-0/data/info.pb
I20260928 15:02:15.195923 22406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:15.199388 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 22339
I20260928 15:02:15.199591 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/minicluster-data/master-0/wal/instance
I20260928 15:02:15.200165 31904 external_mini_cluster.cc:1428] Setting key 41888ef14628c1b9715996055f726657
I20260928 15:02:15.204764 22406 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.40.62" port: 33005 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:02:15 dist-test-slave-j968 krb5kdc[22326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607735, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.62@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.40.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.61@KRBTEST.COM" created.
I20260928 15:02:15.230317 22408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:15.208973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60182 (local address 127.31.40.62:33005)
0928 15:02:15.209914 (+ 941us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:15.209935 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:15.210639 (+ 704us) server_negotiation.cc:407] Connection header received
0928 15:02:15.212773 (+ 2134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:15.212780 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:15.213037 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:15.213262 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:15.214596 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:15.215951 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:15.217374 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:15.217942 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:15.218145 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:15.222253 (+ 4108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:15.222262 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:15.222276 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:15.222307 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:15.225334 (+ 3027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:15.226342 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:15.226349 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:15.226352 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:15.226616 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:15.227120 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:15.227124 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:15.227127 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:15.227776 (+ 649us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:15.228047 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:15.228604 (+ 557us) server_negotiation.cc:299] Negotiation successful
0928 15:02:15.228968 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":668,"thread_start_us":419,"threads_started":1}
I20260928 15:02:15.230343 22409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:15.208973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:33005 (local address 127.0.0.1:60182)
0928 15:02:15.210346 (+ 1373us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:15.210402 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:15.212384 (+ 1982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:15.213271 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:15.213278 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:15.214033 (+ 755us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:15.214379 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:15.214397 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:15.216115 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:15.216126 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:15.217218 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:15.217230 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:15.218180 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:15.218187 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:15.218371 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:15.219061 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:15.219084 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:15.221978 (+ 2894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:15.225492 (+ 3514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:15.225510 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:15.225527 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:15.226172 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:15.226793 (+ 621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:15.226797 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:15.226801 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:15.226984 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:15.228020 (+ 1036us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:15.228026 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:15.228222 (+ 196us) client_negotiation.cc:769] Sending connection context
0928 15:02:15.228439 (+ 217us) client_negotiation.cc:240] Negotiation successful
0928 15:02:15.229186 (+ 747us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1100,"mutex_wait_us":175,"thread_start_us":866,"threads_started":1}
I20260928 15:02:15.233685 22406 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.40.61" port: 45263 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:02:15.235087 22409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:15.234558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.61:45263 (local address 127.0.0.1:41176)
0928 15:02:15.234780 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:15.234927 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.61:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260928 15:02:15.235364 22356 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.40.61:45263: connect: Connection refused (error 111)
I20260928 15:02:15.237112 22409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:15.236647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.61:45263 (local address 127.0.0.1:41184)
0928 15:02:15.236832 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:15.236927 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.61:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 15:02:15.237419 22406 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.40.61:45263: Network error: Client connection negotiation failed: client connection to 127.31.40.61:45263: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.61.keytab.
Entry for principal kudu/127.31.40.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.61.keytab.
I20260928 15:02:15.264173 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.61:45263
--webserver_interface=127.31.40.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.61.keytab
--principal=kudu/127.31.40.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:38035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.40.62:33005,127.31.40.61:45263,127.31.40.60:42911
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:02:15.286782 22406 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.40.61" port: 45263 } attempt: 1
I20260928 15:02:15.288231 22409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:15.287645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.61:45263 (local address 127.0.0.1:41192)
0928 15:02:15.287952 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:15.288076 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.61:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260928 15:02:15.289358 22409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:15.288962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.61:45263 (local address 127.0.0.1:41202)
0928 15:02:15.289113 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:15.289193 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.61:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 15:02:15.289541 22406 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.40.61:45263: Network error: Client connection negotiation failed: client connection to 127.31.40.61:45263: connect: Connection refused (error 111)
I20260928 15:02:15.357834 22406 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.40.61" port: 45263 } attempt: 2
I20260928 15:02:15.359079 22409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:15.358629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.61:45263 (local address 127.0.0.1:41210)
0928 15:02:15.358837 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:15.358943 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.61:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260928 15:02:15.360164 22409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:15.359728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.61:45263 (local address 127.0.0.1:41218)
0928 15:02:15.359919 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:15.360008 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.61:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 15:02:15.360432 22406 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.40.61:45263: Network error: Client connection negotiation failed: client connection to 127.31.40.61:45263: connect: Connection refused (error 111)
I20260928 15:02:15.451691 22406 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.40.61" port: 45263 } attempt: 3
I20260928 15:02:15.453052 22409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:15.452557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.61:45263 (local address 127.0.0.1:41230)
0928 15:02:15.452813 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:15.452908 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.61:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260928 15:02:15.454084 22409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:15.453718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.61:45263 (local address 127.0.0.1:41240)
0928 15:02:15.453885 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:15.453951 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.61:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260928 15:02:15.454258 22406 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.40.61:45263: Network error: Client connection negotiation failed: client connection to 127.31.40.61:45263: connect: Connection refused (error 111)
I20260928 15:02:15.597558 22406 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.40.61" port: 45263 } attempt: 4
I20260928 15:02:15.598981 22409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:15.598393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.61:45263 (local address 127.0.0.1:41246)
0928 15:02:15.598695 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:15.598818 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.61:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
I20260928 15:02:15.600086 22409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:15.599653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.61:45263 (local address 127.0.0.1:41258)
0928 15:02:15.599820 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:15.599959 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.61:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260928 15:02:15.600313 22406 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.40.61:45263: Network error: Client connection negotiation failed: client connection to 127.31.40.61:45263: connect: Connection refused (error 111)
W20260928 15:02:15.606032 22413 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 15:02:15.606382 22413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:15.606506 22413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:15.606653 22413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:15.614529 22413 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:02:15.614615 22413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:15.614717 22413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:15.614786 22413 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:02:15.614873 22413 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:02:15.624878 22413 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:38035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.40.62:33005,127.31.40.61:45263,127.31.40.60:42911
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.61.keytab
--principal=kudu/127.31.40.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.61:45263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.40.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:02:15.626171 22413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:15.627501 22413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:15.639349 22418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:15.639621 22421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:15.639556 22419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:15.640331 22413 server_base.cc:1034] running on GCE node
I20260928 15:02:15.641108 22413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:15.642539 22413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:15.643784 22413 hybrid_clock.cc:648] HybridClock initialized: now 1790607735643723 us; error 49 us; skew 500 ppm
I20260928 15:02:15.644138 22413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:02:15 dist-test-slave-j968 krb5kdc[22326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607735, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:15.647233 22413 init.cc:377] Logged in from keytab as kudu/127.31.40.61@KRBTEST.COM (short username kudu)
I20260928 15:02:15.648952 22413 webserver.cc:460] Webserver started at http://127.31.40.61:39623/ using document root <none> and password file <none>
I20260928 15:02:15.649678 22413 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:15.649767 22413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:15.650108 22413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:15.653211 22413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/minicluster-data/master-1/data/instance:
uuid: "e5bc29b771f34d95b2280a7bd48673c6"
format_stamp: "Formatted at 2026-09-28 15:02:15 on dist-test-slave-j968"
server_key: "0b084ecbbac0467b25685d4c20e015a9"
server_key_iv: "db894f3ed5bcf6391af4ea60ed29b8c9"
server_key_version: "encryptionkey@0"
I20260928 15:02:15.654060 22413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/minicluster-data/master-1/wal/instance:
uuid: "e5bc29b771f34d95b2280a7bd48673c6"
format_stamp: "Formatted at 2026-09-28 15:02:15 on dist-test-slave-j968"
server_key: "0b084ecbbac0467b25685d4c20e015a9"
server_key_iv: "db894f3ed5bcf6391af4ea60ed29b8c9"
server_key_version: "encryptionkey@0"
I20260928 15:02:15.659309 22413 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 15:02:15.663589 22428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:15.665158 22413 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 15:02:15.665355 22413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/minicluster-data/master-1/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/minicluster-data/master-1/wal
uuid: "e5bc29b771f34d95b2280a7bd48673c6"
format_stamp: "Formatted at 2026-09-28 15:02:15 on dist-test-slave-j968"
server_key: "0b084ecbbac0467b25685d4c20e015a9"
server_key_iv: "db894f3ed5bcf6391af4ea60ed29b8c9"
server_key_version: "encryptionkey@0"
I20260928 15:02:15.665602 22413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:15.694321 22413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:15.698732 22413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:15.699038 22413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:15.719107 22413 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.61:45263
I20260928 15:02:15.719129 22479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.61:45263 every 8 connection(s)
I20260928 15:02:15.720623 22413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/minicluster-data/master-1/data/info.pb
I20260928 15:02:15.721596 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 22413
I20260928 15:02:15.721800 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/minicluster-data/master-1/wal/instance
I20260928 15:02:15.722218 31904 external_mini_cluster.cc:1428] Setting key 212264e190ea6c510f4277660aca3f83
I20260928 15:02:15.725050 22480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:15.732724 22480 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.40.62" port: 33005 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.31.40.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.40.60@KRBTEST.COM" created.
Sep 28 15:02:15 dist-test-slave-j968 krb5kdc[22326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607735, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.61@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 15:02:15.755640 22483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:15.735775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60194 (local address 127.31.40.62:33005)
0928 15:02:15.736397 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:15.736404 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:15.737597 (+ 1193us) server_negotiation.cc:407] Connection header received
0928 15:02:15.739797 (+ 2200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:15.739805 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:15.739939 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:15.740074 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:15.741776 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:15.743088 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:15.744829 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:15.745547 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:15.745702 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:15.749645 (+ 3943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:15.749650 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:15.749653 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:15.749693 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:15.752525 (+ 2832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:15.753292 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:15.753297 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:15.753301 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:15.753428 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:15.753848 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:15.753853 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:15.753856 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:15.754142 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:15.754294 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:15.755018 (+ 724us) server_negotiation.cc:299] Negotiation successful
0928 15:02:15.755240 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":319,"threads_started":1}
I20260928 15:02:15.757200 22484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:15.736457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:33005 (local address 127.0.0.1:60194)
0928 15:02:15.737366 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:15.737430 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:15.739474 (+ 2044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:15.740313 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:15.740322 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:15.740919 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:15.741528 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:15.741546 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:15.743285 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:15.743290 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:15.744665 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:15.744679 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:15.745845 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:15.745851 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:15.746062 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:15.746819 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:15.746848 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:15.749384 (+ 2536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:15.752668 (+ 3284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:15.752678 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:15.752696 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:15.753163 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:15.753538 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:15.753543 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:15.753547 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:15.753727 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:15.754270 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:15.754278 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:15.754665 (+ 387us) client_negotiation.cc:769] Sending connection context
0928 15:02:15.754938 (+ 273us) client_negotiation.cc:240] Negotiation successful
0928 15:02:15.755474 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":384,"threads_started":1}
I20260928 15:02:15.759131 22480 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.40.61" port: 45263 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:02:15 dist-test-slave-j968 krb5kdc[22326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607735, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.61@KRBTEST.COM for kudu/127.31.40.61@KRBTEST.COM
I20260928 15:02:15.776741 22484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:15.760037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.61:45263 (local address 127.0.0.1:41262)
0928 15:02:15.760211 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:15.760241 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:15.760324 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:15.761529 (+ 1205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:15.761533 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:15.762053 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:15.762194 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:15.762211 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:15.763626 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:15.763633 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:15.764568 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:15.764581 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:15.765461 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:15.765467 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:15.765644 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:15.766387 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:15.766414 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:15.768601 (+ 2187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:15.773996 (+ 5395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:15.774000 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:15.774007 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:15.774367 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:15.774889 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:15.774893 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:15.774895 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:15.775044 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:15.776003 (+ 959us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:15.776006 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:15.776129 (+ 123us) client_negotiation.cc:769] Sending connection context
0928 15:02:15.776236 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 15:02:15.776474 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 15:02:15.777253 22486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:15.760256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41262 (local address 127.31.40.61:45263)
0928 15:02:15.761036 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:15.761043 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:15.761078 (+ 35us) server_negotiation.cc:407] Connection header received
0928 15:02:15.761127 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:15.761133 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:15.761346 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:15.761531 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:15.762383 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:15.763488 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:15.764730 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:15.765303 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:15.765452 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:15.768976 (+ 3524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:15.768995 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:15.769019 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:15.769069 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:15.773781 (+ 4712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:15.774575 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:15.774588 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:15.774592 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:15.774769 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:15.775187 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:15.775202 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:15.775206 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:15.775796 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:15.776120 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:15.776514 (+ 394us) server_negotiation.cc:299] Negotiation successful
0928 15:02:15.776841 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":372,"threads_started":1}
I20260928 15:02:15.780021 22480 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.40.60" port: 42911 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:02:15.781199 22484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:15.780637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:42911 (local address 127.0.0.1:60792)
0928 15:02:15.780849 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:15.781070 (+ 221us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.60:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260928 15:02:15.781446 22431 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.40.60:42911: connect: Connection refused (error 111)
I20260928 15:02:15.783161 22484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:15.782840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:42911 (local address 127.0.0.1:60798)
0928 15:02:15.782972 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:15.783063 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.60:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 15:02:15.783378 22480 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.40.60:42911: Network error: Client connection negotiation failed: client connection to 127.31.40.60:42911: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.40.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.60.keytab.
Entry for principal kudu/127.31.40.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.60.keytab.
I20260928 15:02:15.788030 31904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
/tmp/dist-test-taskBx3nvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.60:42911
--webserver_interface=127.31.40.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.60.keytab
--principal=kudu/127.31.40.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.40.20:38035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.40.62:33005,127.31.40.61:45263,127.31.40.60:42911
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:02:15.832756 22480 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.40.60" port: 42911 } attempt: 1
I20260928 15:02:15.834193 22484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:15.833631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:42911 (local address 127.0.0.1:60804)
0928 15:02:15.833884 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:15.834013 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.60:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260928 15:02:15.835238 22484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:15.834835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:42911 (local address 127.0.0.1:60814)
0928 15:02:15.835009 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:15.835080 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.60:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 15:02:15.835436 22480 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.40.60:42911: Network error: Client connection negotiation failed: client connection to 127.31.40.60:42911: connect: Connection refused (error 111)
I20260928 15:02:15.899608 22406 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.40.61" port: 45263 } attempt: 5
I20260928 15:02:15.903641 22480 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.40.60" port: 42911 } attempt: 2
I20260928 15:02:15.904590 22484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:15.904165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:42911 (local address 127.0.0.1:60820)
0928 15:02:15.904338 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:15.904445 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.60:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 15:02:15.905740 22484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:15.905318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:42911 (local address 127.0.0.1:60824)
0928 15:02:15.905479 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:15.905577 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.60:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260928 15:02:15.906081 22480 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.40.60:42911: Network error: Client connection negotiation failed: client connection to 127.31.40.60:42911: connect: Connection refused (error 111)
Sep 28 15:02:15 dist-test-slave-j968 krb5kdc[22326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607735, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.62@KRBTEST.COM for kudu/127.31.40.61@KRBTEST.COM
I20260928 15:02:15.914776 22486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:15.900564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41264 (local address 127.31.40.61:45263)
0928 15:02:15.900758 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:15.900763 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:15.900838 (+ 75us) server_negotiation.cc:407] Connection header received
0928 15:02:15.901027 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:15.901031 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:15.901115 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:15.901200 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:15.902363 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:15.903244 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:15.905047 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:15.905658 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:15.905807 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:15.909213 (+ 3406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:15.909217 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:15.909220 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:15.909251 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:15.912070 (+ 2819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:15.912761 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:15.912767 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:15.912775 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:15.912892 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:15.913277 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:15.913283 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:15.913286 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:15.913573 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:15.913728 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:15.914222 (+ 494us) server_negotiation.cc:299] Negotiation successful
0928 15:02:15.914438 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 15:02:15.914776 22409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:15.900444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.61:45263 (local address 127.0.0.1:41264)
0928 15:02:15.900731 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:15.900759 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:15.900910 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:15.901277 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:15.901282 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:15.901881 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:15.902168 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:15.902186 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:15.903409 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:15.903417 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:15.904889 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:15.904905 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:15.905823 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:15.905829 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:15.906030 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:15.906652 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:15.906675 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:15.909001 (+ 2326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:15.912227 (+ 3226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:15.912233 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:15.912237 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:15.912605 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:15.913007 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:15.913013 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:15.913016 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:15.913166 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:15.913704 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:15.913709 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:15.913918 (+ 209us) client_negotiation.cc:769] Sending connection context
0928 15:02:15.914101 (+ 183us) client_negotiation.cc:240] Negotiation successful
0928 15:02:15.914395 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260928 15:02:15.915843 22406 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.40.60" port: 42911 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:02:15.916930 22409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:15.916529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:42911 (local address 127.0.0.1:60834)
0928 15:02:15.916690 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:15.916769 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.60:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 15:02:15.917843 22409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:15.917530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:42911 (local address 127.0.0.1:60846)
0928 15:02:15.917675 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:15.917754 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.60:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 15:02:15.918064 22406 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.40.60:42911: Network error: Client connection negotiation failed: client connection to 127.31.40.60:42911: connect: Connection refused (error 111)
I20260928 15:02:15.969367 22406 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.40.60" port: 42911 } attempt: 1
I20260928 15:02:15.970651 22409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:15.970129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:42911 (local address 127.0.0.1:60858)
0928 15:02:15.970347 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:15.970458 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.60:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260928 15:02:15.971727 22409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:15.971371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:42911 (local address 127.0.0.1:60862)
0928 15:02:15.971553 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:15.971621 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.60:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260928 15:02:15.971968 22406 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.40.60:42911: Network error: Client connection negotiation failed: client connection to 127.31.40.60:42911: connect: Connection refused (error 111)
I20260928 15:02:15.997334 22480 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.40.60" port: 42911 } attempt: 3
I20260928 15:02:15.998579 22484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:15.998091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:42911 (local address 127.0.0.1:60868)
0928 15:02:15.998321 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:15.998420 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.60:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260928 15:02:15.999614 22484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:15.999193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:42911 (local address 127.0.0.1:60884)
0928 15:02:15.999428 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:15.999500 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.60:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
W20260928 15:02:15.999814 22480 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.40.60:42911: Network error: Client connection negotiation failed: client connection to 127.31.40.60:42911: connect: Connection refused (error 111)
I20260928 15:02:16.040220 22406 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.40.60" port: 42911 } attempt: 2
I20260928 15:02:16.041571 22409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:16.041042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:42911 (local address 127.0.0.1:60892)
0928 15:02:16.041305 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:16.041399 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.60:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260928 15:02:16.042702 22409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:16.042316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:42911 (local address 127.0.0.1:60904)
0928 15:02:16.042489 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:16.042577 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.60:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260928 15:02:16.042935 22406 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.40.60:42911: Network error: Client connection negotiation failed: client connection to 127.31.40.60:42911: connect: Connection refused (error 111)
I20260928 15:02:16.143030 22480 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.40.60" port: 42911 } attempt: 4
I20260928 15:02:16.144546 22484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:16.143803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:42911 (local address 127.0.0.1:60910)
0928 15:02:16.144172 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:16.144304 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.60:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260928 15:02:16.145649 22484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:16.145264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:42911 (local address 127.0.0.1:60920)
0928 15:02:16.145446 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:16.145528 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.60:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260928 15:02:16.145919 22480 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.40.60:42911: Network error: Client connection negotiation failed: client connection to 127.31.40.60:42911: connect: Connection refused (error 111)
I20260928 15:02:16.149125 22406 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.40.60" port: 42911 } attempt: 3
I20260928 15:02:16.150190 22409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:16.149697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:42911 (local address 127.0.0.1:60930)
0928 15:02:16.149888 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:16.149980 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.60:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 15:02:16.151204 22409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:16.150808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:42911 (local address 127.0.0.1:60938)
0928 15:02:16.150989 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:16.151072 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.40.60:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260928 15:02:16.151436 22406 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.40.60:42911: Network error: Client connection negotiation failed: client connection to 127.31.40.60:42911: connect: Connection refused (error 111)
W20260928 15:02:16.156632 22488 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 15:02:16.157030 22488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:16.157150 22488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:16.157286 22488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:16.165297 22488 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:02:16.165422 22488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:16.165542 22488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:16.165628 22488 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:02:16.165742 22488 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:02:16.176375 22488 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.40.20:38035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.40.62:33005,127.31.40.61:45263,127.31.40.60:42911
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/krb5kdc/kudu_127.31.40.60.keytab
--principal=kudu/127.31.40.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.40.60:42911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.40.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:02:16.177752 22488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:16.179095 22488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:16.191401 22493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:16.191427 22496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:16.191829 22494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:16.192746 22488 server_base.cc:1034] running on GCE node
I20260928 15:02:16.193612 22488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:16.195662 22488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:16.197022 22488 hybrid_clock.cc:648] HybridClock initialized: now 1790607736196933 us; error 58 us; skew 500 ppm
I20260928 15:02:16.197430 22488 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:02:16 dist-test-slave-j968 krb5kdc[22326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607736, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:16.200757 22488 init.cc:377] Logged in from keytab as kudu/127.31.40.60@KRBTEST.COM (short username kudu)
I20260928 15:02:16.202219 22488 webserver.cc:460] Webserver started at http://127.31.40.60:33673/ using document root <none> and password file <none>
I20260928 15:02:16.202863 22488 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:16.202950 22488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:16.203222 22488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:16.206296 22488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/minicluster-data/master-2/data/instance:
uuid: "bbc8d918dbb34502aba6e39e2d714fec"
format_stamp: "Formatted at 2026-09-28 15:02:16 on dist-test-slave-j968"
server_key: "d5f7c03fd394bbcd93b3fe9a0889bdda"
server_key_iv: "8e888cae4897d40f8e3083aad97aac9a"
server_key_version: "encryptionkey@0"
I20260928 15:02:16.207046 22488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/minicluster-data/master-2/wal/instance:
uuid: "bbc8d918dbb34502aba6e39e2d714fec"
format_stamp: "Formatted at 2026-09-28 15:02:16 on dist-test-slave-j968"
server_key: "d5f7c03fd394bbcd93b3fe9a0889bdda"
server_key_iv: "8e888cae4897d40f8e3083aad97aac9a"
server_key_version: "encryptionkey@0"
I20260928 15:02:16.212316 22488 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260928 15:02:16.215941 22503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:16.217327 22488 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:02:16.217483 22488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/minicluster-data/master-2/data,/tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/minicluster-data/master-2/wal
uuid: "bbc8d918dbb34502aba6e39e2d714fec"
format_stamp: "Formatted at 2026-09-28 15:02:16 on dist-test-slave-j968"
server_key: "d5f7c03fd394bbcd93b3fe9a0889bdda"
server_key_iv: "8e888cae4897d40f8e3083aad97aac9a"
server_key_version: "encryptionkey@0"
I20260928 15:02:16.217670 22488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:16.236678 22488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:16.241525 22488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:16.241786 22488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:16.261791 22488 rpc_server.cc:307] RPC server started. Bound to: 127.31.40.60:42911
I20260928 15:02:16.261838 22554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.40.60:42911 every 8 connection(s)
I20260928 15:02:16.263255 22488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/minicluster-data/master-2/data/info.pb
I20260928 15:02:16.266667 31904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu as pid 22488
I20260928 15:02:16.266871 31904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBx3nvY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607365362981-31904-0/minicluster-data/master-2/wal/instance
I20260928 15:02:16.267459 31904 external_mini_cluster.cc:1428] Setting key ffddea15f9be91e7b999d4b022a397f0
I20260928 15:02:16.268116 22555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:16.277225 22555 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.40.62" port: 33005 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:02:16 dist-test-slave-j968 krb5kdc[22326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.60@KRBTEST.COM
Sep 28 15:02:16 dist-test-slave-j968 krb5kdc[22326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607736, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.60@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
I20260928 15:02:16.291368 22558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:16.269524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60942 (local address 127.31.40.60:42911)
0928 15:02:16.270404 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:16.270428 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:16.270472 (+ 44us) server_negotiation.cc:407] Connection header received
0928 15:02:16.272169 (+ 1697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:16.272193 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:16.272638 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:16.273092 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:16.274587 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:16.276004 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:16.277254 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:16.277977 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:16.278179 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:16.283134 (+ 4955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:16.283141 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:16.283152 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:16.283188 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:16.286474 (+ 3286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:16.287245 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:16.287252 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:16.287258 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:16.287433 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:16.287899 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:16.287906 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:16.287909 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:16.288700 (+ 791us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:16.289069 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:16.289576 (+ 507us) server_negotiation.cc:299] Negotiation successful
0928 15:02:16.290032 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":360,"threads_started":1}
I20260928 15:02:16.296456 22559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:16.279769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60204 (local address 127.31.40.62:33005)
0928 15:02:16.280380 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:16.280388 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:16.280930 (+ 542us) server_negotiation.cc:407] Connection header received
0928 15:02:16.281130 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:16.281137 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:16.281220 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:16.281307 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:16.282894 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:16.283580 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:16.285494 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:16.285923 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:16.286069 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:16.289616 (+ 3547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:16.289622 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:16.289625 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:16.289663 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:16.293033 (+ 3370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:16.293930 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:16.293935 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:16.293938 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:16.294148 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:16.294597 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:16.294602 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:16.294604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:16.294980 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:16.295136 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:16.295726 (+ 590us) server_negotiation.cc:299] Negotiation successful
0928 15:02:16.295984 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":306,"threads_started":1}
I20260928 15:02:16.297528 22560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:16.279725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.62:33005 (local address 127.0.0.1:60204)
0928 15:02:16.280766 (+ 1041us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:16.280814 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:16.280983 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:16.281409 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:16.281416 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:16.282301 (+ 885us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:16.282671 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:16.282685 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:16.284001 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:16.284007 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:16.285328 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:16.285342 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:16.286115 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:16.286122 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:16.286304 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:16.286938 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:16.286956 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:16.289402 (+ 2446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:16.293176 (+ 3774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:16.293188 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:16.293196 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:16.293756 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:16.294285 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:16.294290 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:16.294293 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:16.294464 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:16.295124 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:16.295132 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:16.295352 (+ 220us) client_negotiation.cc:769] Sending connection context
0928 15:02:16.296663 (+ 1311us) client_negotiation.cc:240] Negotiation successful
0928 15:02:16.297129 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":789,"thread_start_us":636,"threads_started":1}
I20260928 15:02:16.299150 22555 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.40.61" port: 45263 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:02:16 dist-test-slave-j968 krb5kdc[22326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.62@KRBTEST.COM
Sep 28 15:02:16 dist-test-slave-j968 krb5kdc[22326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607736, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.60@KRBTEST.COM for kudu/127.31.40.61@KRBTEST.COM
I20260928 15:02:16.312835 22561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:16.295181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60212 (local address 127.31.40.62:33005)
0928 15:02:16.295801 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:16.295805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:16.295823 (+ 18us) server_negotiation.cc:407] Connection header received
0928 15:02:16.295860 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:16.295912 (+ 52us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:16.295979 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:16.296063 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:16.297751 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:16.299005 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:16.301063 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:16.301551 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:16.301719 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:16.306833 (+ 5114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:16.306839 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:16.306843 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:16.306877 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:16.310026 (+ 3149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:16.310702 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:16.310708 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:16.310712 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:16.310854 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:16.311241 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:16.311250 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:16.311252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:16.311529 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:16.311805 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:16.312174 (+ 369us) server_negotiation.cc:299] Negotiation successful
0928 15:02:16.312411 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":291,"threads_started":1}
I20260928 15:02:16.312947 22560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:16.300058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.61:45263 (local address 127.0.0.1:41268)
0928 15:02:16.300345 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:16.300359 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:16.300449 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:16.300798 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:16.300802 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:16.301189 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:16.301330 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:16.301341 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:16.302330 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:16.302337 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:16.303228 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:16.303240 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:16.304203 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:16.304207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:16.304344 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:16.304939 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:16.304955 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:16.307241 (+ 2286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:16.310453 (+ 3212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:16.310458 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:16.310463 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:16.310875 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:16.311268 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:16.311273 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:16.311277 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:16.311431 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:16.311991 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:16.311996 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:16.312154 (+ 158us) client_negotiation.cc:769] Sending connection context
0928 15:02:16.312294 (+ 140us) client_negotiation.cc:240] Negotiation successful
0928 15:02:16.312590 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195}
I20260928 15:02:16.312975 22486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:16.300062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41268 (local address 127.31.40.61:45263)
0928 15:02:16.300236 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:16.300240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:16.300449 (+ 209us) server_negotiation.cc:407] Connection header received
0928 15:02:16.300534 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:16.300540 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:16.300635 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:16.300758 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:16.301502 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:16.302190 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:16.303377 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:16.303866 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:16.304009 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:16.307451 (+ 3442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:16.307456 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:16.307459 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:16.307493 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:16.310210 (+ 2717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:16.311006 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:16.311011 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:16.311016 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:16.311152 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:16.311539 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:16.311546 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:16.311549 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:16.311811 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:16.312030 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:16.312396 (+ 366us) server_negotiation.cc:299] Negotiation successful
0928 15:02:16.312602 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 15:02:16.314612 22555 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.40.60" port: 42911 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:02:16 dist-test-slave-j968 krb5kdc[22326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607736, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.60@KRBTEST.COM for kudu/127.31.40.60@KRBTEST.COM
Sep 28 15:02:16 dist-test-slave-j968 krb5kdc[22326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.40.61@KRBTEST.COM
I20260928 15:02:16.328637 22406 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.40.60" port: 42911 } attempt: 4
I20260928 15:02:16.329470 22560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:16.315537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:42911 (local address 127.0.0.1:60948)
0928 15:02:16.315683 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:16.315700 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:16.315793 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:16.316097 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:16.316101 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:16.316500 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:16.316633 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:16.316642 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:16.317579 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:16.317586 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:16.318449 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:16.318466 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:16.319169 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:16.319175 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:16.319363 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:16.320006 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:16.320024 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:16.322684 (+ 2660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:16.326455 (+ 3771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:16.326462 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:16.326466 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:16.327009 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:16.327549 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:16.327554 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:16.327558 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:16.327729 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:16.328340 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:16.328345 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:16.328535 (+ 190us) client_negotiation.cc:769] Sending connection context
0928 15:02:16.328729 (+ 194us) client_negotiation.cc:240] Negotiation successful
0928 15:02:16.329071 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 15:02:16.330668 22558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:16.315621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60948 (local address 127.31.40.60:42911)
0928 15:02:16.315785 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:16.315790 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:16.315806 (+ 16us) server_negotiation.cc:407] Connection header received
0928 15:02:16.315902 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:16.315908 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:16.315971 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:16.316074 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:16.316777 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:16.317457 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:16.318593 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:16.319017 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:16.319154 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:16.322869 (+ 3715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:16.322875 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:16.322880 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:16.322918 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:16.326281 (+ 3363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:16.327173 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:16.327180 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:16.327185 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:16.327360 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:16.327847 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:16.327851 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:16.327853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:16.328205 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:16.328335 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:16.328852 (+ 517us) server_negotiation.cc:299] Negotiation successful
0928 15:02:16.330232 (+ 1380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1068,"server-negotiator.queue_time_us":55}
I20260928 15:02:16.336404 22555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbc8d918dbb34502aba6e39e2d714fec: Bootstrap starting.
I20260928 15:02:16.336503 22486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:16.314976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41282 (local address 127.31.40.61:45263)
0928 15:02:16.315383 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:16.315392 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:16.315414 (+ 22us) server_negotiation.cc:407] Connection header received
0928 15:02:16.315470 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:16.315475 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:16.315550 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:16.315655 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:16.317359 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:16.318232 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:16.319608 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:16.320160 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:16.320312 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:16.325445 (+ 5133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:16.325451 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:16.325454 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:16.325490 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:16.329082 (+ 3592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:16.330134 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:16.330141 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:16.330146 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:16.330337 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:16.330871 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:16.330878 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:16.330881 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:16.331293 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:16.331601 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:16.335837 (+ 4236us) server_negotiation.cc:299] Negotiation successful
0928 15:02:16.336121 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272}
Sep 28 15:02:16 dist-test-slave-j968 krb5kdc[22326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607735, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.62@KRBTEST.COM for kudu/127.31.40.60@KRBTEST.COM
I20260928 15:02:16.340819 22555 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbc8d918dbb34502aba6e39e2d714fec: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:16.342034 31904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:02:16.342329 22555 log.cc:824] T 00000000000000000000000000000000 P bbc8d918dbb34502aba6e39e2d714fec: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:16.345830 22555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbc8d918dbb34502aba6e39e2d714fec: No bootstrap required, opened a new log
I20260928 15:02:16.345185 22381 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "f8eb0cd40dc74a50bd040a59f1372fbe"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60212
I20260928 15:02:16.350425 22409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:16.329435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:42911 (local address 127.0.0.1:60954)
0928 15:02:16.329749 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:16.329770 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:16.329901 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:16.330501 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:16.330509 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:16.331053 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:16.331293 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:16.331307 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:16.332981 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:16.332986 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:16.334024 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:16.334037 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:16.335014 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:16.335019 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:16.335158 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:16.335796 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:16.335819 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:16.338736 (+ 2917us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:16.342816 (+ 4080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:16.342824 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:16.342831 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:16.343347 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:16.343783 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:16.343789 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:16.343793 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:16.344039 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:16.344701 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:16.344706 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:16.349471 (+ 4765us) client_negotiation.cc:769] Sending connection context
0928 15:02:16.349669 (+ 198us) client_negotiation.cc:240] Negotiation successful
0928 15:02:16.350016 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172}
I20260928 15:02:16.350669 22562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:16.329471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60954 (local address 127.31.40.60:42911)
0928 15:02:16.330181 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:16.330188 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:16.330210 (+ 22us) server_negotiation.cc:407] Connection header received
0928 15:02:16.330257 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:16.330261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:16.330335 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:16.330461 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:16.331511 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:16.332835 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:16.334169 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:16.334842 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:16.334966 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:16.338981 (+ 4015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:16.338988 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:16.338992 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:16.339030 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:16.342644 (+ 3614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:16.343497 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:16.343503 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:16.343507 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:16.343657 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:16.344163 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:16.344169 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:16.344172 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:16.344545 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:16.344706 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:16.349897 (+ 5191us) server_negotiation.cc:299] Negotiation successful
0928 15:02:16.350163 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":673,"threads_started":1}
I20260928 15:02:16.354933 22406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8eb0cd40dc74a50bd040a59f1372fbe: Bootstrap starting.
I20260928 15:02:16.357081 22555 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbc8d918dbb34502aba6e39e2d714fec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8eb0cd40dc74a50bd040a59f1372fbe" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 33005 } } peers { permanent_uuid: "e5bc29b771f34d95b2280a7bd48673c6" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 45263 } } peers { permanent_uuid: "bbc8d918dbb34502aba6e39e2d714fec" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 42911 } }
I20260928 15:02:16.357493 22555 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbc8d918dbb34502aba6e39e2d714fec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbc8d918dbb34502aba6e39e2d714fec, State: Initialized, Role: FOLLOWER
I20260928 15:02:16.358017 22555 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbc8d918dbb34502aba6e39e2d714fec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8eb0cd40dc74a50bd040a59f1372fbe" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 33005 } } peers { permanent_uuid: "e5bc29b771f34d95b2280a7bd48673c6" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 45263 } } peers { permanent_uuid: "bbc8d918dbb34502aba6e39e2d714fec" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 42911 } }
I20260928 15:02:16.358477 22406 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8eb0cd40dc74a50bd040a59f1372fbe: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:16.359045 22564 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbc8d918dbb34502aba6e39e2d714fec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8eb0cd40dc74a50bd040a59f1372fbe" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 33005 } } peers { permanent_uuid: "e5bc29b771f34d95b2280a7bd48673c6" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 45263 } } peers { permanent_uuid: "bbc8d918dbb34502aba6e39e2d714fec" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 42911 } } }
I20260928 15:02:16.359470 22564 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbc8d918dbb34502aba6e39e2d714fec [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:02:16.359812 22406 log.cc:824] T 00000000000000000000000000000000 P f8eb0cd40dc74a50bd040a59f1372fbe: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:16.360352 22555 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbc8d918dbb34502aba6e39e2d714fec [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:02:16.363139 22406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8eb0cd40dc74a50bd040a59f1372fbe: No bootstrap required, opened a new log
W20260928 15:02:16.370723 22576 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bbc8d918dbb34502aba6e39e2d714fec: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 15:02:16.370949 22576 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 15:02:16.370929 22406 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8eb0cd40dc74a50bd040a59f1372fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8eb0cd40dc74a50bd040a59f1372fbe" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 33005 } } peers { permanent_uuid: "e5bc29b771f34d95b2280a7bd48673c6" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 45263 } } peers { permanent_uuid: "bbc8d918dbb34502aba6e39e2d714fec" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 42911 } }
I20260928 15:02:16.371359 22406 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8eb0cd40dc74a50bd040a59f1372fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8eb0cd40dc74a50bd040a59f1372fbe, State: Initialized, Role: FOLLOWER
I20260928 15:02:16.371805 22406 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8eb0cd40dc74a50bd040a59f1372fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8eb0cd40dc74a50bd040a59f1372fbe" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 33005 } } peers { permanent_uuid: "e5bc29b771f34d95b2280a7bd48673c6" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 45263 } } peers { permanent_uuid: "bbc8d918dbb34502aba6e39e2d714fec" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 42911 } }
I20260928 15:02:16.372790 22578 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8eb0cd40dc74a50bd040a59f1372fbe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8eb0cd40dc74a50bd040a59f1372fbe" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 33005 } } peers { permanent_uuid: "e5bc29b771f34d95b2280a7bd48673c6" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 45263 } } peers { permanent_uuid: "bbc8d918dbb34502aba6e39e2d714fec" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 42911 } } }
I20260928 15:02:16.373225 22578 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8eb0cd40dc74a50bd040a59f1372fbe [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:02:16.373986 22406 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8eb0cd40dc74a50bd040a59f1372fbe [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:02:16.374095 22381 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8eb0cd40dc74a50bd040a59f1372fbe [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 15:02:16.374302 22381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8eb0cd40dc74a50bd040a59f1372fbe [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:16.377955 22381 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8eb0cd40dc74a50bd040a59f1372fbe [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8eb0cd40dc74a50bd040a59f1372fbe" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 33005 } } peers { permanent_uuid: "e5bc29b771f34d95b2280a7bd48673c6" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 45263 } } peers { permanent_uuid: "bbc8d918dbb34502aba6e39e2d714fec" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 42911 } }
I20260928 15:02:16.368185 22488 master_runner.cc:186] Error getting master registration for 127.31.40.62:33005: OK, instance_id { permanent_uuid: "f8eb0cd40dc74a50bd040a59f1372fbe" instance_seqno: 1790607735184557 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 15:02:16.378953 22488 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 15:02:16.380319 22381 leader_election.cc:288] T 00000000000000000000000000000000 P f8eb0cd40dc74a50bd040a59f1372fbe [CANDIDATE]: Term 1 election: Requested vote from peers e5bc29b771f34d95b2280a7bd48673c6 (127.31.40.61:45263), bbc8d918dbb34502aba6e39e2d714fec (127.31.40.60:42911)
I20260928 15:02:16.381436 22530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f8eb0cd40dc74a50bd040a59f1372fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "bbc8d918dbb34502aba6e39e2d714fec"
I20260928 15:02:16.382009 22530 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbc8d918dbb34502aba6e39e2d714fec [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:16.386777 22530 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bbc8d918dbb34502aba6e39e2d714fec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8eb0cd40dc74a50bd040a59f1372fbe in term 1.
I20260928 15:02:16.387682 22357 leader_election.cc:302] T 00000000000000000000000000000000 P f8eb0cd40dc74a50bd040a59f1372fbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bbc8d918dbb34502aba6e39e2d714fec, f8eb0cd40dc74a50bd040a59f1372fbe; no voters:
I20260928 15:02:16.388433 22578 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8eb0cd40dc74a50bd040a59f1372fbe [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:16.391669 22578 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8eb0cd40dc74a50bd040a59f1372fbe [term 1 LEADER]: Becoming Leader. State: Replica: f8eb0cd40dc74a50bd040a59f1372fbe, State: Running, Role: LEADER
I20260928 15:02:16.392431 22578 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8eb0cd40dc74a50bd040a59f1372fbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8eb0cd40dc74a50bd040a59f1372fbe" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 33005 } } peers { permanent_uuid: "e5bc29b771f34d95b2280a7bd48673c6" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 45263 } } peers { permanent_uuid: "bbc8d918dbb34502aba6e39e2d714fec" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 42911 } }
I20260928 15:02:16.405265 22586 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8eb0cd40dc74a50bd040a59f1372fbe [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8eb0cd40dc74a50bd040a59f1372fbe. Latest consensus state: current_term: 1 leader_uuid: "f8eb0cd40dc74a50bd040a59f1372fbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8eb0cd40dc74a50bd040a59f1372fbe" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 33005 } } peers { permanent_uuid: "e5bc29b771f34d95b2280a7bd48673c6" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 45263 } } peers { permanent_uuid: "bbc8d918dbb34502aba6e39e2d714fec" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 42911 } } }
I20260928 15:02:16.405741 22586 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8eb0cd40dc74a50bd040a59f1372fbe [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:16.407137 22592 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260928 15:02:16.422657 22360 tablet.cc:2369] T 00000000000000000000000000000000 P f8eb0cd40dc74a50bd040a59f1372fbe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:02:16.445154 22480 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.40.60" port: 42911 } attempt: 5
Sep 28 15:02:16 dist-test-slave-j968 krb5kdc[22326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607735, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.40.61@KRBTEST.COM for kudu/127.31.40.60@KRBTEST.COM
I20260928 15:02:16.459357 22592 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:02:16.465175 22484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:16.445805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.40.60:42911 (local address 127.0.0.1:60978)
0928 15:02:16.446190 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:16.446223 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:16.446354 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:16.447079 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:16.447086 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:16.447613 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:16.447823 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:16.447835 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:16.449381 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:16.449387 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:16.450833 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:16.450850 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:16.452076 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:16.452082 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:16.452310 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:16.453705 (+ 1395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:16.453731 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:16.457013 (+ 3282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:16.462111 (+ 5098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:16.462117 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:16.462121 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:16.462654 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:16.463118 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:16.463123 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:16.463128 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:16.463359 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:16.464097 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:16.464102 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:16.464273 (+ 171us) client_negotiation.cc:769] Sending connection context
0928 15:02:16.464440 (+ 167us) client_negotiation.cc:240] Negotiation successful
0928 15:02:16.464744 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260}
I20260928 15:02:16.466154 22558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:16.446565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60978 (local address 127.31.40.60:42911)
0928 15:02:16.446742 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:16.446750 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:16.446770 (+ 20us) server_negotiation.cc:407] Connection header received
0928 15:02:16.446825 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:16.446833 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:16.446937 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:16.447039 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:16.448064 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:16.449161 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:16.451011 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:16.451690 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:16.451818 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:16.457306 (+ 5488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:16.457314 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:16.457323 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:16.457373 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:16.461947 (+ 4574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:16.462799 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:16.462804 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:16.462808 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:16.462970 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:16.463496 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:16.463501 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:16.463505 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:16.463857 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:16.464030 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:16.465573 (+ 1543us) server_negotiation.cc:299] Negotiation successful
0928 15:02:16.465789 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 15:02:16.466670 22486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:16.411498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41284 (local address 127.31.40.61:45263)
0928 15:02:16.411740 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:16.411749 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:16.415052 (+ 3303us) server_negotiation.cc:407] Connection header received
0928 15:02:16.416186 (+ 1134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:16.416196 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:16.416308 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:16.416398 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:16.427263 (+ 10865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:16.428533 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:16.436140 (+ 7607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:16.436798 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:16.436888 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:16.451352 (+ 14464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:16.451361 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:16.451368 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:16.451423 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:16.457577 (+ 6154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:16.463700 (+ 6123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:16.463708 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:16.463714 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:16.463897 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:16.464337 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:16.464343 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:16.464347 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:16.464730 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:16.465001 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:16.465822 (+ 821us) server_negotiation.cc:299] Negotiation successful
0928 15:02:16.466074 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 15:02:16.468269 22562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:16.414627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60966 (local address 127.31.40.60:42911)
0928 15:02:16.414873 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:16.414883 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:16.422955 (+ 8072us) server_negotiation.cc:407] Connection header received
0928 15:02:16.426055 (+ 3100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:16.426063 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:16.426212 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:16.426308 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:16.429090 (+ 2782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:16.430410 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:16.438093 (+ 7683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:16.438958 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:16.439103 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:16.452754 (+ 13651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:16.452761 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:16.452766 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:16.452804 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:16.458542 (+ 5738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:16.460014 (+ 1472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:16.460020 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:16.460024 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:16.460186 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:16.462938 (+ 2752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:16.462945 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:16.462947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:16.463418 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:16.465257 (+ 1839us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:16.466170 (+ 913us) server_negotiation.cc:299] Negotiation successful
0928 15:02:16.467585 (+ 1415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 15:02:16.470287 22561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:16.412800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60228 (local address 127.31.40.62:33005)
0928 15:02:16.413021 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:16.413027 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:16.423387 (+ 10360us) server_negotiation.cc:407] Connection header received
0928 15:02:16.423858 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:16.423879 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:16.423985 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:16.424068 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:16.430824 (+ 6756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:16.432060 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:16.436024 (+ 3964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:16.436755 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:16.436869 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:16.455145 (+ 18276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:16.455159 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:16.455167 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:16.455222 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:16.458980 (+ 3758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:16.460675 (+ 1695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:16.460681 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:16.460684 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:16.460911 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:16.462063 (+ 1152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:16.462070 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:16.462075 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:16.462448 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:16.469482 (+ 7034us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:16.469645 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 15:02:16.469900 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 15:02:16.474128 22480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5bc29b771f34d95b2280a7bd48673c6: Bootstrap starting.
I20260928 15:02:16.482821 22480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5bc29b771f34d95b2280a7bd48673c6: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:16.487936 22480 log.cc:824] T 00000000000000000000000000000000 P e5bc29b771f34d95b2280a7bd48673c6: Log is configured to *not* fsync() on all Append() calls
W20260928 15:02:16.493964 22509 tablet.cc:2369] T 00000000000000000000000000000000 P bbc8d918dbb34502aba6e39e2d714fec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:02:16.495857 22480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5bc29b771f34d95b2280a7bd48673c6: No bootstrap required, opened a new log
I20260928 15:02:16.507527 22480 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5bc29b771f34d95b2280a7bd48673c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8eb0cd40dc74a50bd040a59f1372fbe" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 33005 } } peers { permanent_uuid: "e5bc29b771f34d95b2280a7bd48673c6" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 45263 } } peers { permanent_uuid: "bbc8d918dbb34502aba6e39e2d714fec" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 42911 } }
I20260928 15:02:16.507983 22480 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5bc29b771f34d95b2280a7bd48673c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5bc29b771f34d95b2280a7bd48673c6, State: Initialized, Role: FOLLOWER
I20260928 15:02:16.508527 22480 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5bc29b771f34d95b2280a7bd48673c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8eb0cd40dc74a50bd040a59f1372fbe" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 33005 } } peers { permanent_uuid: "e5bc29b771f34d95b2280a7bd48673c6" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 45263 } } peers { permanent_uuid: "bbc8d918dbb34502aba6e39e2d714fec" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 42911 } }
I20260928 15:02:16.509790 22603 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5bc29b771f34d95b2280a7bd48673c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8eb0cd40dc74a50bd040a59f1372fbe" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 33005 } } peers { permanent_uuid: "e5bc29b771f34d95b2280a7bd48673c6" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 45263 } } peers { permanent_uuid: "bbc8d918dbb34502aba6e39e2d714fec" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 42911 } } }
I20260928 15:02:16.510530 22603 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5bc29b771f34d95b2280a7bd48673c6 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:02:16.513229 22530 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bbc8d918dbb34502aba6e39e2d714fec [term 1 FOLLOWER]: Refusing update from remote peer f8eb0cd40dc74a50bd040a59f1372fbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:02:16.497073 22339 master_runner.cc:186] Error getting master registration for 127.31.40.61:45263: OK, instance_id { permanent_uuid: "e5bc29b771f34d95b2280a7bd48673c6" instance_seqno: 1790607735713518 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 15:02:16.513962 22339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 15:02:16.515700 22586 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f8eb0cd40dc74a50bd040a59f1372fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbc8d918dbb34502aba6e39e2d714fec" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 42911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:02:16.518394 22454 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5bc29b771f34d95b2280a7bd48673c6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:16.519846 22455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f8eb0cd40dc74a50bd040a59f1372fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e5bc29b771f34d95b2280a7bd48673c6"
I20260928 15:02:16.522792 22454 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e5bc29b771f34d95b2280a7bd48673c6 [term 1 FOLLOWER]: Refusing update from remote peer f8eb0cd40dc74a50bd040a59f1372fbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:02:16.526846 22455 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e5bc29b771f34d95b2280a7bd48673c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8eb0cd40dc74a50bd040a59f1372fbe in term 1.
I20260928 15:02:16.527736 22586 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f8eb0cd40dc74a50bd040a59f1372fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bc29b771f34d95b2280a7bd48673c6" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 45263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:02:16.530768 22480 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5bc29b771f34d95b2280a7bd48673c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:02:16.548017 22564 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbc8d918dbb34502aba6e39e2d714fec [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8eb0cd40dc74a50bd040a59f1372fbe. Latest consensus state: current_term: 1 leader_uuid: "f8eb0cd40dc74a50bd040a59f1372fbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8eb0cd40dc74a50bd040a59f1372fbe" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 33005 } } peers { permanent_uuid: "e5bc29b771f34d95b2280a7bd48673c6" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 45263 } } peers { permanent_uuid: "bbc8d918dbb34502aba6e39e2d714fec" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 42911 } } }
I20260928 15:02:16.548436 22564 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbc8d918dbb34502aba6e39e2d714fec [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:02:16.554344 22564 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbc8d918dbb34502aba6e39e2d714fec [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f8eb0cd40dc74a50bd040a59f1372fbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8eb0cd40dc74a50bd040a59f1372fbe" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 33005 } } peers { permanent_uuid: "e5bc29b771f34d95b2280a7bd48673c6" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 45263 } } peers { permanent_uuid: "bbc8d918dbb34502aba6e39e2d714fec" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 42911 } } }
I20260928 15:02:16.554697 22564 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbc8d918dbb34502aba6e39e2d714fec [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:02:16.568377 22578 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8eb0cd40dc74a50bd040a59f1372fbe [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f8eb0cd40dc74a50bd040a59f1372fbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8eb0cd40dc74a50bd040a59f1372fbe" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 33005 } } peers { permanent_uuid: "e5bc29b771f34d95b2280a7bd48673c6" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 45263 } } peers { permanent_uuid: "bbc8d918dbb34502aba6e39e2d714fec" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 42911 } } }
I20260928 15:02:16.568843 22578 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8eb0cd40dc74a50bd040a59f1372fbe [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:16.575210 22592 catalog_manager.cc:1347] Generated new cluster ID: 6e46059d527845238e6b1f27ba47b6eb
I20260928 15:02:16.575351 22592 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:02:16.600723 22603 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5bc29b771f34d95b2280a7bd48673c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8eb0cd40dc74a50bd040a59f1372fbe. Latest consensus state: current_term: 1 leader_uuid: "f8eb0cd40dc74a50bd040a59f1372fbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8eb0cd40dc74a50bd040a59f1372fbe" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 33005 } } peers { permanent_uuid: "e5bc29b771f34d95b2280a7bd48673c6" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 45263 } } peers { permanent_uuid: "bbc8d918dbb34502aba6e39e2d714fec" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 42911 } } }
I20260928 15:02:16.601187 22603 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5bc29b771f34d95b2280a7bd48673c6 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:02:16.605525 22604 mvcc.cc:205] Tried to move back new op lower bound from 7334329288745713664 to 7334329288270213120. Current Snapshot: MvccSnapshot[applied={T|T < 7334329288745713664 or (T in {7334329288745713664})}]
I20260928 15:02:16.606676 22592 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:02:16.609206 22592 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:02:16.613236 22586 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8eb0cd40dc74a50bd040a59f1372fbe [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f8eb0cd40dc74a50bd040a59f1372fbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8eb0cd40dc74a50bd040a59f1372fbe" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 33005 } } peers { permanent_uuid: "e5bc29b771f34d95b2280a7bd48673c6" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 45263 } } peers { permanent_uuid: "bbc8d918dbb34502aba6e39e2d714fec" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 42911 } } }
I20260928 15:02:16.613536 22586 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8eb0cd40dc74a50bd040a59f1372fbe [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:16.630125 22603 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5bc29b771f34d95b2280a7bd48673c6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f8eb0cd40dc74a50bd040a59f1372fbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8eb0cd40dc74a50bd040a59f1372fbe" member_type: VOTER last_known_addr { host: "127.31.40.62" port: 33005 } } peers { permanent_uuid: "e5bc29b771f34d95b2280a7bd48673c6" member_type: VOTER last_known_addr { host: "127.31.40.61" port: 45263 } } peers { permanent_uuid: "bbc8d918dbb34502aba6e39e2d714fec" member_type: VOTER last_known_addr { host: "127.31.40.60" port: 42911 } } }
I20260928 15:02:16.630509 22603 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5bc29b771f34d95b2280a7bd48673c6 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:02:16.641583 22592 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8eb0cd40dc74a50bd040a59f1372fbe: Generated new TSK 0
I20260928 15:02:16.643092 22592 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:02:16.683173 22622 catalog_manager.cc:1259] Loaded cluster ID: 6e46059d527845238e6b1f27ba47b6eb
I20260928 15:02:16.683383 22622 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e5bc29b771f34d95b2280a7bd48673c6: loading cluster ID for follower catalog manager: success
I20260928 15:02:16.687413 22622 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e5bc29b771f34d95b2280a7bd48673c6: acquiring CA information for follower catalog manager: success
I20260928 15:02:16.694648 22622 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e5bc29b771f34d95b2280a7bd48673c6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 15:02:16.785619 22561 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:02:16.763912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60242 (local address 127.31.40.62:33005)
0928 15:02:16.764143 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:16.764151 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:16.782869 (+ 18718us) server_negotiation.cc:407] Connection header received
0928 15:02:16.783466 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:16.783476 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:16.783578 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:16.783658 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:16.785301 (+ 1643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60242: BlockingRecv error: recv got EOF from 127.0.0.1:60242 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260928 15:02:16.790110 22558 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:02:16.787317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60984 (local address 127.31.40.60:42911)
0928 15:02:16.787499 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:16.787505 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:16.788194 (+ 689us) server_negotiation.cc:407] Connection header received
0928 15:02:16.788457 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:16.788463 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:16.788549 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:16.788621 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:16.789843 (+ 1222us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60984: BlockingRecv error: recv got EOF from 127.0.0.1:60984 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260928 15:02:16.791567 22486 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:02:16.765612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41290 (local address 127.31.40.61:45263)
0928 15:02:16.765820 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:16.765827 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:16.786283 (+ 20456us) server_negotiation.cc:407] Connection header received
0928 15:02:16.786451 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:16.786458 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:16.786551 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:16.786669 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:16.791305 (+ 4636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41290: BlockingRecv error: recv got EOF from 127.0.0.1:41290 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 15:02:16.852902 22486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:16.833008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41296 (local address 127.31.40.61:45263)
0928 15:02:16.833228 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:16.833236 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:16.843948 (+ 10712us) server_negotiation.cc:407] Connection header received
0928 15:02:16.844588 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:16.844596 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:16.844849 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:16.844998 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:16.845803 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:16.847190 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:16.849738 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:16.850456 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:16.850560 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:16.851355 (+ 795us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:16.852073 (+ 718us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:16.852194 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:16.852439 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 15:02:16.852558 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260928 15:02:16.897081 22558 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:02:16.887241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60986 (local address 127.31.40.60:42911)
0928 15:02:16.887431 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:16.887439 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:16.894773 (+ 7334us) server_negotiation.cc:407] Connection header received
0928 15:02:16.895355 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:16.895362 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:16.895521 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:16.895631 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:16.896801 (+ 1170us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60986: BlockingRecv error: recv got EOF from 127.0.0.1:60986 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 15:02:16.947835 22486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:16.929442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41302 (local address 127.31.40.61:45263)
0928 15:02:16.929637 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:16.929646 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:16.940410 (+ 10764us) server_negotiation.cc:407] Connection header received
0928 15:02:16.940981 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:16.940988 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:16.941110 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:16.941179 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:16.941958 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:16.943162 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:16.945324 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:16.946014 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:16.946123 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:16.946890 (+ 767us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:16.947068 (+ 178us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:16.947135 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:16.947376 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 15:02:16.947468 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260928 15:02:16.992159 22558 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:02:16.982160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60994 (local address 127.31.40.60:42911)
0928 15:02:16.982395 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:16.982402 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:16.990040 (+ 7638us) server_negotiation.cc:407] Connection header received
0928 15:02:16.990608 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:16.990617 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:16.990721 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:16.990814 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:16.991895 (+ 1081us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60994: BlockingRecv error: recv got EOF from 127.0.0.1:60994 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 15:02:17.048949 22486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:17.030681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41318 (local address 127.31.40.61:45263)
0928 15:02:17.030887 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:17.030893 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:17.041559 (+ 10666us) server_negotiation.cc:407] Connection header received
0928 15:02:17.042180 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:17.042187 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:17.042296 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:17.042374 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:17.043083 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:17.044271 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:17.046385 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:17.047056 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:17.047183 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:17.047939 (+ 756us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:17.048139 (+ 200us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:17.048215 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:17.048478 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 15:02:17.048574 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260928 15:02:17.070565 22558 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:02:17.062587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60998 (local address 127.31.40.60:42911)
0928 15:02:17.062805 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:17.062811 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:17.068635 (+ 5824us) server_negotiation.cc:407] Connection header received
0928 15:02:17.068869 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:17.068877 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:17.069005 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:17.069134 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:17.070256 (+ 1122us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60998: BlockingRecv error: recv got EOF from 127.0.0.1:60998 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 15:02:17.085896 22486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:17.078692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41326 (local address 127.31.40.61:45263)
0928 15:02:17.078967 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:17.078973 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:17.079557 (+ 584us) server_negotiation.cc:407] Connection header received
0928 15:02:17.079777 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:17.079783 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:17.079902 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:17.080007 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:17.080501 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:17.081735 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:17.083713 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:17.084474 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:17.084617 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:17.085111 (+ 494us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:17.085255 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:17.085376 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:17.085522 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 15:02:17.085610 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
W20260928 15:02:17.095184 22558 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:02:17.093505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32772 (local address 127.31.40.60:42911)
0928 15:02:17.093753 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:17.093759 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:17.093780 (+ 21us) server_negotiation.cc:407] Connection header received
0928 15:02:17.093855 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:17.093861 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:17.093952 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:17.094066 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:17.094956 (+ 890us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32772: BlockingRecv error: recv got EOF from 127.0.0.1:32772 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
I20260928 15:02:17.114813 22486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:17.107503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41340 (local address 127.31.40.61:45263)
0928 15:02:17.107719 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:17.107725 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:17.108360 (+ 635us) server_negotiation.cc:407] Connection header received
0928 15:02:17.108594 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:17.108600 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:17.108689 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:17.108789 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:17.109285 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:17.110419 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:17.112556 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:17.113334 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:17.113444 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:17.113968 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:17.114115 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:17.114223 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:17.114384 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 15:02:17.114471 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260928 15:02:17.124372 22558 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:02:17.121967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32788 (local address 127.31.40.60:42911)
0928 15:02:17.122141 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:17.122147 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:17.122673 (+ 526us) server_negotiation.cc:407] Connection header received
0928 15:02:17.122865 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:17.122873 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:17.123014 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:17.123142 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:17.124077 (+ 935us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32788: BlockingRecv error: recv got EOF from 127.0.0.1:32788 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 15:02:17.152107 22486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:17.144572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41342 (local address 127.31.40.61:45263)
0928 15:02:17.144771 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:17.144777 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:17.145750 (+ 973us) server_negotiation.cc:407] Connection header received
0928 15:02:17.146034 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:17.146040 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:17.146131 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:17.146241 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:17.146827 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:17.148066 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:17.150017 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:17.150584 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:17.150722 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:17.151266 (+ 544us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:17.151406 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:17.151490 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:17.151685 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 15:02:17.151782 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260928 15:02:17.165093 22558 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:02:17.162230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32796 (local address 127.31.40.60:42911)
0928 15:02:17.162431 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:17.162438 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:17.163059 (+ 621us) server_negotiation.cc:407] Connection header received
0928 15:02:17.163418 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:17.163425 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:17.163531 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:17.163612 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:17.164854 (+ 1242us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32796: BlockingRecv error: recv got EOF from 127.0.0.1:32796 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 15:02:17.208391 22486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:17.200962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41346 (local address 127.31.40.61:45263)
0928 15:02:17.201199 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:17.201205 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:17.201788 (+ 583us) server_negotiation.cc:407] Connection header received
0928 15:02:17.201997 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:17.202003 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:17.202135 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:17.202237 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:17.202725 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:17.203917 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:17.205970 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:17.206674 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:17.206814 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:17.207328 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:17.207478 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:17.207565 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:17.207767 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 15:02:17.207982 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260928 15:02:17.217561 22558 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:02:17.215464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32806 (local address 127.31.40.60:42911)
0928 15:02:17.215676 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:17.215682 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:17.215964 (+ 282us) server_negotiation.cc:407] Connection header received
0928 15:02:17.216220 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:17.216225 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:17.216302 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:17.216389 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:17.217314 (+ 925us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32806: BlockingRecv error: recv got EOF from 127.0.0.1:32806 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
I20260928 15:02:17.290459 22486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:17.285306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41348 (local address 127.31.40.61:45263)
0928 15:02:17.285514 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:17.285518 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:17.285915 (+ 397us) server_negotiation.cc:407] Connection header received
0928 15:02:17.286099 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:17.286107 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:17.286233 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:17.286326 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:17.286705 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:17.287460 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:17.288795 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:17.289229 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:17.289332 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:17.289697 (+ 365us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:17.289884 (+ 187us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:17.289956 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:17.290165 (+ 209us) server_negotiation.cc:299] Negotiation successful
0928 15:02:17.290238 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260928 15:02:17.298295 22558 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:02:17.296467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32820 (local address 127.31.40.60:42911)
0928 15:02:17.296699 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:17.296705 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:17.296937 (+ 232us) server_negotiation.cc:407] Connection header received
0928 15:02:17.297110 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:17.297114 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:17.297191 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:17.297299 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:17.298073 (+ 774us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32820: BlockingRecv error: recv got EOF from 127.0.0.1:32820 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
I20260928 15:02:17.373134 22576 catalog_manager.cc:1259] Loaded cluster ID: 6e46059d527845238e6b1f27ba47b6eb
I20260928 15:02:17.373245 22576 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bbc8d918dbb34502aba6e39e2d714fec: loading cluster ID for follower catalog manager: success
I20260928 15:02:17.376109 22576 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bbc8d918dbb34502aba6e39e2d714fec: acquiring CA information for follower catalog manager: success
I20260928 15:02:17.377957 22576 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bbc8d918dbb34502aba6e39e2d714fec: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 15:02:17.436228 22486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:17.430382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41350 (local address 127.31.40.61:45263)
0928 15:02:17.430646 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:17.430652 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:17.431066 (+ 414us) server_negotiation.cc:407] Connection header received
0928 15:02:17.431237 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:17.431241 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:17.431317 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:17.431407 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:17.432014 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:17.432979 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:17.434463 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:17.434911 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:17.435006 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:17.435472 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:17.435586 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:17.435655 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:17.435891 (+ 236us) server_negotiation.cc:299] Negotiation successful
0928 15:02:17.435962 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260928 15:02:17.450937 22558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:17.442317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32826 (local address 127.31.40.60:42911)
0928 15:02:17.443201 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:17.443220 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:17.443245 (+ 25us) server_negotiation.cc:407] Connection header received
0928 15:02:17.443323 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:17.443331 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:17.443551 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:17.443706 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:17.444226 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:17.445504 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:17.447902 (+ 2398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:17.448686 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:17.448835 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:17.449420 (+ 585us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:17.450205 (+ 785us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:17.450366 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:17.450609 (+ 243us) server_negotiation.cc:299] Negotiation successful
0928 15:02:17.450704 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":710}
I20260928 15:02:17.465382 22486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:17.457344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41352 (local address 127.31.40.61:45263)
0928 15:02:17.457582 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:17.457590 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:17.458677 (+ 1087us) server_negotiation.cc:407] Connection header received
0928 15:02:17.458889 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:17.458893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:17.458961 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:17.459057 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:17.459522 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:17.460552 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:17.463093 (+ 2541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:17.463900 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:17.464049 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:17.464598 (+ 549us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:17.464741 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:17.464839 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:17.464994 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 15:02:17.465084 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 15:02:17.467479 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:17.457004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60252 (local address 127.31.40.62:33005)
0928 15:02:17.460267 (+ 3263us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:17.460274 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:17.460301 (+ 27us) server_negotiation.cc:407] Connection header received
0928 15:02:17.460410 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:17.460415 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:17.460656 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:17.460847 (+ 191us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:17.461758 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:17.462978 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:17.465090 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:17.465711 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:17.465840 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:17.466420 (+ 580us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:17.466781 (+ 361us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:17.466869 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:17.467057 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 15:02:17.467130 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3119,"thread_start_us":3018,"threads_started":1}
I20260928 15:02:17.469065 22558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:17.457699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32840 (local address 127.31.40.60:42911)
0928 15:02:17.457860 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:17.457864 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:17.461640 (+ 3776us) server_negotiation.cc:407] Connection header received
0928 15:02:17.463352 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:17.463356 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:17.463423 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:17.463528 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:17.464243 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:17.465202 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:17.467276 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:17.467797 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:17.467984 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:17.468388 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:17.468508 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:17.468609 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:17.468721 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 15:02:17.468809 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 15:02:17.472630 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 22339
I20260928 15:02:17.473981 22401 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:02:17.586086 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 22339
I20260928 15:02:17.608517 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 22413
I20260928 15:02:17.609776 22475 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:02:17.721127 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 22413
I20260928 15:02:17.742558 31904 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu pid 22488
I20260928 15:02:17.743840 22550 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:02:17.855240 31904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBx3nvY/build/asan/bin/kudu with pid 22488
2026-09-28T15:02:17Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4562 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9130 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (372457 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST